2024-04-25T19:04:48.614 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T19:04:48.615 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302 --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} --owner scheduled_pdonnell@teuthology --verbose -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/orig.config.yaml 2024-04-25T19:04:48.661 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T19:04:48.746 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} email: pdonnell@redhat.com first_in_suite: false job_id: '7673302' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - unmatched fragstat - unmatched rstat - was unreadable, recreating it now - Scrub error on inode - Metadata damage detected - MDS_FAILED - MDS_DAMAGE sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJmvhjPUNA6O+Zm951C9UJ8iIOyB9VdjG3iONnVc4gi+6f6U2s0ccHrA7ficl6kRUwDV54Q4Lv4VVGUye0/hiI= smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+uYEmH8wMWDuHceKaHtf+8QcIMfKsL70h7iFRIWpcEu0VUuc4Jxstes/4KeBy7tp+7jcuds47ka/VTX79sfao= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_recovery_pool teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T19:04:48.746 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T19:04:48.747 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T19:04:48.747 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T19:04:48.748 INFO:teuthology.task.internal:Checking packages... 2024-04-25T19:04:48.771 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T19:04:48.771 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T19:04:48.771 INFO:teuthology.packaging:ref: None 2024-04-25T19:04:48.771 INFO:teuthology.packaging:tag: None 2024-04-25T19:04:48.771 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T19:04:48.771 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:04:48.772 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T19:04:48.911 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T19:04:48.913 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T19:04:48.919 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T19:04:48.919 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T19:04:48.925 INFO:teuthology.task.internal:Saving configuration 2024-04-25T19:04:48.942 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T19:04:48.947 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T19:04:48.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 19:00:20.374095', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJmvhjPUNA6O+Zm951C9UJ8iIOyB9VdjG3iONnVc4gi+6f6U2s0ccHrA7ficl6kRUwDV54Q4Lv4VVGUye0/hiI='} 2024-04-25T19:04:48.996 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 19:00:20.375835', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+uYEmH8wMWDuHceKaHtf+8QcIMfKsL70h7iFRIWpcEu0VUuc4Jxstes/4KeBy7tp+7jcuds47ka/VTX79sfao='} 2024-04-25T19:04:48.996 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T19:04:49.002 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-25T19:04:49.002 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-25T19:04:49.002 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T19:04:49.066 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7e149d3f70>, signals=[15]) 2024-04-25T19:04:49.067 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T19:04:49.073 INFO:teuthology.task.internal:Opening connections... 2024-04-25T19:04:49.073 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:04:49.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:04:49.188 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:04:49.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:04:49.261 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T19:04:49.268 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-04-25T19:04:49.286 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-04-25T19:04:49.286 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T19:04:49.344 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-04-25T19:04:49.345 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T19:04:49.345 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T19:04:49.345 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T19:04:49.345 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-04-25T19:04:49.375 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-04-25T19:04:49.394 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-04-25T19:04:49.394 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-04-25T19:04:49.452 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-04-25T19:04:49.452 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-04-25T19:04:49.452 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-04-25T19:04:49.452 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-04-25T19:04:49.452 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T19:04:49.453 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T19:04:49.453 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-04-25T19:04:49.475 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T19:04:49.485 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T19:04:49.491 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T19:04:49.491 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-04-25T19:04:49.494 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-04-25T19:04:49.511 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T19:04:49.517 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T19:04:49.517 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-04-25T19:04:49.551 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-04-25T19:04:49.575 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T19:04:49.622 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T19:04:49.636 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T19:04:49.636 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-04-25T19:04:49.636 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-04-25T19:04:49.636 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T19:04:49.637 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T19:04:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:04:49.654 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-25T19:04:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:04:49.655 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-04-25T19:04:49.711 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-04-25T19:04:49.712 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-437.el9.x86_64 2024-04-25T19:04:49.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-04-25T19:04:49.713 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-437.el9.x86_64 2024-04-25T19:04:49.713 INFO:teuthology.task.kernel:Running kernel on smithi053: 5.14.0-437.el9.x86_64 2024-04-25T19:04:49.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum install -y kernel 2024-04-25T19:04:52.025 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T19:04:52.304 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.0 MB 00:01 2024-04-25T19:04:54.193 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-25T19:04:54.630 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-25T19:04:58.835 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.2 MB 00:00 2024-04-25T19:04:59.223 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-25T19:05:00.525 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-25T19:05:01.679 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-25T19:05:03.585 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 21 MB 00:02 2024-04-25T19:05:03.916 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-04-25T19:05:08.801 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-25T19:05:09.205 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-04-25T19:05:10.766 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T19:05:10.766 INFO:teuthology.orchestra.run.smithi053.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T19:05:10.845 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-25T19:05:10.846 INFO:teuthology.orchestra.run.smithi053.stdout:Nothing to do. 2024-04-25T19:05:10.846 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-04-25T19:05:10.957 DEBUG:teuthology.orchestra.run.smithi053:> echo no | sudo yum reinstall kernel || true 2024-04-25T19:05:11.162 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T19:05:11.162 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T19:05:11.254 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-25T19:05:11.255 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-25T19:05:11.255 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-25T19:05:11.375 DEBUG:teuthology.orchestra.run.smithi028:> echo no | sudo yum reinstall kernel || true 2024-04-25T19:05:11.478 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 07:05:08 PM UTC. 2024-04-25T19:05:11.610 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout:Reinstalling: 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:05:11.621 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 255 k 2024-04-25T19:05:11.622 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 0 2024-04-25T19:05:11.622 INFO:teuthology.orchestra.run.smithi053.stderr:Operation aborted. 2024-04-25T19:05:11.681 INFO:teuthology.orchestra.run.smithi053.stdout:Is this ok [y/N]: 2024-04-25T19:05:11.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum reinstall -y kernel || true 2024-04-25T19:05:11.898 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:02 ago on Thu 25 Apr 2024 07:05:09 PM UTC. 2024-04-25T19:05:12.034 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-25T19:05:12.052 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T19:05:12.052 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-25T19:05:12.053 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T19:05:12.053 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalling: 2024-04-25T19:05:12.053 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T19:05:12.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:05:12.053 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-25T19:05:12.053 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T19:05:12.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:05:12.053 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 255 k 2024-04-25T19:05:12.054 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 0 2024-04-25T19:05:12.054 INFO:teuthology.orchestra.run.smithi028.stderr:Operation aborted. 2024-04-25T19:05:12.116 INFO:teuthology.orchestra.run.smithi028.stdout:Is this ok [y/N]: 2024-04-25T19:05:12.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum reinstall -y kernel || true 2024-04-25T19:05:12.199 INFO:teuthology.orchestra.run.smithi053.stdout:Last metadata expiration check: 0:00:04 ago on Thu 25 Apr 2024 07:05:08 PM UTC. 2024-04-25T19:05:12.319 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-25T19:05:12.320 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-25T19:05:12.320 INFO:teuthology.orchestra.run.smithi053.stdout: Package Architecture Version Repository Size 2024-04-25T19:05:12.320 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-25T19:05:12.320 INFO:teuthology.orchestra.run.smithi053.stdout:Reinstalling: 2024-04-25T19:05:12.320 INFO:teuthology.orchestra.run.smithi053.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T19:05:12.320 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:05:12.320 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-04-25T19:05:12.321 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-04-25T19:05:12.321 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:05:12.321 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 255 k 2024-04-25T19:05:12.321 INFO:teuthology.orchestra.run.smithi053.stdout:Installed size: 0 2024-04-25T19:05:12.321 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-04-25T19:05:12.632 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 07:05:09 PM UTC. 2024-04-25T19:05:12.752 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalling: 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 255 k 2024-04-25T19:05:12.753 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 0 2024-04-25T19:05:12.754 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-25T19:05:13.070 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-437.el9.x86_64.rpm 654 kB/s | 255 kB 00:00 2024-04-25T19:05:13.070 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:05:13.070 INFO:teuthology.orchestra.run.smithi053.stdout:Total 340 kB/s | 255 kB 00:00 2024-04-25T19:05:13.089 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-04-25T19:05:13.283 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-04-25T19:05:13.283 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-04-25T19:05:13.289 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-04-25T19:05:13.289 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-04-25T19:05:13.316 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-437.el9.x86_64.rpm 768 kB/s | 255 kB 00:00 2024-04-25T19:05:13.316 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:05:13.317 INFO:teuthology.orchestra.run.smithi028.stdout:Total 452 kB/s | 255 kB 00:00 2024-04-25T19:05:13.333 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-25T19:05:13.452 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-04-25T19:05:13.514 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-25T19:05:13.514 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-25T19:05:13.519 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-25T19:05:13.520 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-25T19:05:13.529 INFO:teuthology.orchestra.run.smithi053.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T19:05:13.530 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:05:13.666 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-25T19:05:13.717 INFO:teuthology.orchestra.run.smithi028.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T19:05:13.717 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:05:14.205 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:05:14.205 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T19:05:14.421 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:05:14.422 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T19:05:14.905 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:05:14.905 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:05:14.905 INFO:teuthology.orchestra.run.smithi053.stdout:Reinstalled: 2024-04-25T19:05:14.905 INFO:teuthology.orchestra.run.smithi053.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T19:05:14.905 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:05:14.906 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-04-25T19:05:15.092 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T19:05:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T19:05:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:05:15.117 INFO:teuthology.orchestra.run.smithi028.stdout:Reinstalled: 2024-04-25T19:05:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T19:05:15.117 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:05:15.117 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-25T19:05:15.140 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T19:05:15.140 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T19:05:15.140 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T19:05:15.141 DEBUG:teuthology.parallel:result is None 2024-04-25T19:05:15.287 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T19:05:15.334 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T19:05:15.335 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T19:05:15.335 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T19:05:15.335 DEBUG:teuthology.parallel:result is None 2024-04-25T19:05:15.335 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T19:05:15.344 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T19:05:15.344 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T19:05:15.346 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T19:05:15.362 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T19:05:15.368 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T19:05:15.374 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T19:05:15.374 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T19:05:15.402 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T19:05:15.428 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T19:05:15.435 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T19:05:15.435 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T19:05:15.466 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T19:05:15.494 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:05:15.498 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:05:15.506 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:05:15.511 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T19:05:15.513 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T19:05:15.519 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T19:05:15.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T19:05:15.556 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T19:05:15.580 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T19:05:15.588 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T19:05:15.589 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T19:05:15.621 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T19:05:15.636 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:05:15.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:05:15.764 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:05:15.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:05:15.885 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:05:15.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T19:05:15.949 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:05:15.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T19:05:16.039 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:05:16.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T19:05:16.160 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:05:16.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T19:05:16.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-04-25T19:05:16.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-04-25T19:05:16.257 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:05:16.302 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:05:16.711 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T19:05:16.719 INFO:teuthology.task.internal:Starting timer... 2024-04-25T19:05:16.719 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T19:05:16.733 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T19:05:16.741 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T19:05:16.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-04-25T19:05:16.817 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-04-25T19:05:16.819 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-04-25T19:05:16.878 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-04-25T19:05:16.880 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T19:05:16.880 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-04-25T19:05:16.901 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-04-25T19:05:16.902 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-04-25T19:05:16.924 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-04-25T19:05:16.924 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi053.front.sepia.ceph.com': 'permissive'} 2024-04-25T19:05:16.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:05:16.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:05:16.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:05:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:05:16.992 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T19:05:16.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-04-25T19:05:17.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-04-25T19:05:17.057 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T19:05:17.067 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T19:05:17.073 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T19:05:17.215 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T19:05:17.228 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T19:05:17.230 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi053.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T19:08:49.489 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi053.front.sepia.ceph.com')] 2024-04-25T19:08:49.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-25T19:08:49.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:08:49.569 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-25T19:08:49.646 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-25T19:08:49.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-25T19:08:49.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T19:08:49.721 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-04-25T19:08:49.799 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-04-25T19:08:49.799 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T19:08:49.810 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T19:08:49.810 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T19:08:49.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:08:49.813 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T19:08:49.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:08:49.841 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T19:08:49.855 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T19:08:49.869 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T19:08:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-04-25T19:08:49.883 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T19:08:49.890 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-04-25T19:08:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T19:08:49.907 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-04-25T19:08:49.918 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-04-25T19:08:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T19:08:49.932 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T19:08:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T19:08:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:08:49.982 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:08:49.982 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-25T19:08:49.982 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:08:49.982 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:08:49.982 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:08:49.982 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:08:50.000 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:08:50.003 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:08:50.003 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-04-25T19:08:50.004 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:08:50.004 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:08:50.004 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:08:50.004 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:08:50.005 INFO:teuthology.run_tasks:Running task install... 2024-04-25T19:08:50.013 DEBUG:teuthology.task.install:project ceph 2024-04-25T19:08:50.013 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T19:08:50.013 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T19:08:50.013 INFO:teuthology.task.install:Using flavor: default 2024-04-25T19:08:50.020 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T19:08:50.020 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T19:08:50.021 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T19:08:50.021 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:08:50.023 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T19:08:50.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:08:50.240 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T19:08:50.241 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T19:08:50.254 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T19:08:50.255 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T19:08:50.348 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T19:08:50.348 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:08:50.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T19:08:50.374 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T19:08:50.374 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T19:08:50.421 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T19:08:50.422 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:08:50.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T19:08:50.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T19:08:50.447 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T19:08:50.447 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T19:08:50.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T19:08:50.514 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T19:08:50.578 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-04-25T19:08:50.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-04-25T19:08:50.587 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T19:08:50.650 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-04-25T19:08:50.652 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-04-25T19:08:50.814 INFO:teuthology.orchestra.run.smithi028.stdout:48 files removed 2024-04-25T19:08:50.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T19:08:50.885 INFO:teuthology.orchestra.run.smithi053.stdout:48 files removed 2024-04-25T19:08:50.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T19:08:51.535 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 197 kB/s | 84 kB 00:00 2024-04-25T19:08:51.546 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 232 kB/s | 84 kB 00:00 2024-04-25T19:08:51.888 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 39 kB/s | 12 kB 00:00 2024-04-25T19:08:51.969 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-04-25T19:08:52.247 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-04-25T19:08:52.303 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2024-04-25T19:08:53.208 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.0 MB 00:00 2024-04-25T19:08:53.547 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-04-25T19:08:55.584 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-25T19:08:55.715 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-04-25T19:09:00.394 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.2 MB 00:01 2024-04-25T19:09:00.799 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.2 MB 00:01 2024-04-25T19:09:02.043 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-25T19:09:02.439 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-25T19:09:03.252 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-25T19:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 21 MB 00:02 2024-04-25T19:09:08.510 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-25T19:09:10.624 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-25T19:09:10.737 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T19:09:10.738 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T19:09:10.750 INFO:teuthology.orchestra.run.smithi053.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T19:09:10.750 INFO:teuthology.orchestra.run.smithi053.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T19:09:10.751 INFO:teuthology.orchestra.run.smithi053.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T19:09:10.752 INFO:teuthology.orchestra.run.smithi053.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T19:09:10.754 INFO:teuthology.orchestra.run.smithi053.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T19:09:10.756 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T19:09:10.757 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T19:09:10.823 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T19:09:10.833 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T19:09:10.834 INFO:teuthology.orchestra.run.smithi053.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T19:09:10.835 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T19:09:10.836 INFO:teuthology.orchestra.run.smithi053.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T19:09:10.837 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T19:09:10.838 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T19:09:10.839 INFO:teuthology.orchestra.run.smithi053.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T19:09:10.840 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T19:09:10.841 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T19:09:10.842 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T19:09:10.843 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T19:09:10.844 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:09:10.845 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-04-25T19:09:10.845 INFO:teuthology.orchestra.run.smithi053.stdout:======================================================================================= 2024-04-25T19:09:10.845 INFO:teuthology.orchestra.run.smithi053.stdout:Install 148 Packages 2024-04-25T19:09:10.845 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 4 Packages 2024-04-25T19:09:10.845 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:09:10.845 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 214 M 2024-04-25T19:09:10.845 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-04-25T19:09:12.320 INFO:teuthology.orchestra.run.smithi053.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-04-25T19:09:12.877 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T19:09:12.878 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T19:09:12.889 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T19:09:12.890 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T19:09:12.891 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T19:09:12.892 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T19:09:12.893 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T19:09:12.896 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T19:09:12.897 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T19:09:12.964 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-25T19:09:12.973 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T19:09:12.974 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T19:09:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T19:09:12.976 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T19:09:12.977 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T19:09:12.978 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T19:09:12.979 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T19:09:12.980 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T19:09:12.981 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T19:09:12.982 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T19:09:12.983 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T19:09:12.984 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T19:09:12.985 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T19:09:12.986 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-04-25T19:09:12.987 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 4 Packages 2024-04-25T19:09:12.988 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:09:12.988 INFO:teuthology.orchestra.run.smithi053.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 863 kB 00:00 2024-04-25T19:09:12.988 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 214 M 2024-04-25T19:09:12.989 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-25T19:09:13.137 INFO:teuthology.orchestra.run.smithi053.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 930 kB/s | 147 kB 00:00 2024-04-25T19:09:13.562 INFO:teuthology.orchestra.run.smithi053.stdout:(4/152): ceph-common-19.0.0-3291.g994e2f20.el9. 13 MB/s | 20 MB 00:01 2024-04-25T19:09:13.662 INFO:teuthology.orchestra.run.smithi053.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 15 MB/s | 1.5 MB 00:00 2024-04-25T19:09:13.984 INFO:teuthology.orchestra.run.smithi028.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 34 kB/s | 6.6 kB 00:00 2024-04-25T19:09:14.029 INFO:teuthology.orchestra.run.smithi053.stdout:(6/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 13 MB/s | 4.9 MB 00:00 2024-04-25T19:09:14.147 INFO:teuthology.orchestra.run.smithi053.stdout:(7/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 2.2 MB/s | 2.3 MB 00:01 2024-04-25T19:09:14.651 INFO:teuthology.orchestra.run.smithi028.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 863 kB 00:00 2024-04-25T19:09:14.784 INFO:teuthology.orchestra.run.smithi028.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 1.1 MB/s | 147 kB 00:00 2024-04-25T19:09:15.097 INFO:teuthology.orchestra.run.smithi053.stdout:(8/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x86 17 MB/s | 18 MB 00:01 2024-04-25T19:09:15.164 INFO:teuthology.orchestra.run.smithi053.stdout:(9/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-25T19:09:15.189 INFO:teuthology.orchestra.run.smithi053.stdout:(10/152): ceph-selinux-19.0.0-3291.g994e2f20.el 275 kB/s | 25 kB 00:00 2024-04-25T19:09:15.251 INFO:teuthology.orchestra.run.smithi028.stdout:(4/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 3.7 MB/s | 5.4 MB 00:01 2024-04-25T19:09:15.389 INFO:teuthology.orchestra.run.smithi053.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.1 MB/s | 224 kB 00:00 2024-04-25T19:09:15.523 INFO:teuthology.orchestra.run.smithi053.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 203 kB/s | 27 kB 00:00 2024-04-25T19:09:15.559 INFO:teuthology.orchestra.run.smithi028.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 4.9 MB/s | 1.5 MB 00:00 2024-04-25T19:09:15.648 INFO:teuthology.orchestra.run.smithi053.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 254 kB/s | 32 kB 00:00 2024-04-25T19:09:15.886 INFO:teuthology.orchestra.run.smithi028.stdout:(6/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 2.1 MB/s | 2.3 MB 00:01 2024-04-25T19:09:16.293 INFO:teuthology.orchestra.run.smithi028.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 6.6 MB/s | 4.9 MB 00:00 2024-04-25T19:09:16.406 INFO:teuthology.orchestra.run.smithi053.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 997 kB/s | 756 kB 00:00 2024-04-25T19:09:16.673 INFO:teuthology.orchestra.run.smithi053.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 592 kB/s | 158 kB 00:00 2024-04-25T19:09:16.924 INFO:teuthology.orchestra.run.smithi053.stdout:(16/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 3.6 MB/s | 10 MB 00:02 2024-04-25T19:09:16.949 INFO:teuthology.orchestra.run.smithi053.stdout:(17/152): librados-devel-19.0.0-3291.g994e2f20. 461 kB/s | 127 kB 00:00 2024-04-25T19:09:17.227 INFO:teuthology.orchestra.run.smithi028.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 11 MB/s | 10 MB 00:00 2024-04-25T19:09:17.294 INFO:teuthology.orchestra.run.smithi028.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 380 kB/s | 25 kB 00:00 2024-04-25T19:09:17.349 INFO:teuthology.orchestra.run.smithi053.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 1.1 MB/s | 497 kB 00:00 2024-04-25T19:09:17.457 INFO:teuthology.orchestra.run.smithi053.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 417 kB/s | 45 kB 00:00 2024-04-25T19:09:17.649 INFO:teuthology.orchestra.run.smithi053.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 705 kB/s | 135 kB 00:00 2024-04-25T19:09:17.858 INFO:teuthology.orchestra.run.smithi053.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 773 kB/s | 161 kB 00:00 2024-04-25T19:09:18.159 INFO:teuthology.orchestra.run.smithi053.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 1.0 MB/s | 321 kB 00:00 2024-04-25T19:09:18.693 INFO:teuthology.orchestra.run.smithi053.stdout:(23/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 12 MB/s | 43 MB 00:03 2024-04-25T19:09:18.718 INFO:teuthology.orchestra.run.smithi053.stdout:(24/152): python3-rbd-19.0.0-3291.g994e2f20.el9 534 kB/s | 298 kB 00:00 2024-04-25T19:09:18.760 INFO:teuthology.orchestra.run.smithi053.stdout:(25/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.5 MB/s | 100 kB 00:00 2024-04-25T19:09:18.927 INFO:teuthology.orchestra.run.smithi053.stdout:(26/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 414 kB/s | 86 kB 00:00 2024-04-25T19:09:19.127 INFO:teuthology.orchestra.run.smithi053.stdout:(27/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 8.3 MB/s | 3.0 MB 00:00 2024-04-25T19:09:19.152 INFO:teuthology.orchestra.run.smithi053.stdout:(28/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 752 kB/s | 169 kB 00:00 2024-04-25T19:09:19.186 INFO:teuthology.orchestra.run.smithi053.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3291.g 586 kB/s | 34 kB 00:00 2024-04-25T19:09:19.253 INFO:teuthology.orchestra.run.smithi053.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.4 MB/s | 142 kB 00:00 2024-04-25T19:09:19.603 INFO:teuthology.orchestra.run.smithi053.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 8.5 MB/s | 3.5 MB 00:00 2024-04-25T19:09:19.687 INFO:teuthology.orchestra.run.smithi053.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.9 MB/s | 249 kB 00:00 2024-04-25T19:09:19.761 INFO:teuthology.orchestra.run.smithi053.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 660 kB/s | 49 kB 00:00 2024-04-25T19:09:19.837 INFO:teuthology.orchestra.run.smithi053.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 219 kB/s | 16 kB 00:00 2024-04-25T19:09:19.929 INFO:teuthology.orchestra.run.smithi053.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 2.9 MB/s | 273 kB 00:00 2024-04-25T19:09:20.046 INFO:teuthology.orchestra.run.smithi053.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 6.4 MB/s | 762 kB 00:00 2024-04-25T19:09:20.088 INFO:teuthology.orchestra.run.smithi053.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 438 kB/s | 18 kB 00:00 2024-04-25T19:09:20.247 INFO:teuthology.orchestra.run.smithi053.stdout:(38/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 1.6 MB/s | 5.2 MB 00:03 2024-04-25T19:09:20.377 INFO:teuthology.orchestra.run.smithi028.stdout:(10/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 14 MB/s | 43 MB 00:03 2024-04-25T19:09:20.427 INFO:teuthology.orchestra.run.smithi028.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 4.4 MB/s | 224 kB 00:00 2024-04-25T19:09:20.469 INFO:teuthology.orchestra.run.smithi028.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 649 kB/s | 27 kB 00:00 2024-04-25T19:09:20.472 INFO:teuthology.orchestra.run.smithi053.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.5 MB/s | 803 kB 00:00 2024-04-25T19:09:20.497 INFO:teuthology.orchestra.run.smithi053.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 196 kB/s | 80 kB 00:00 2024-04-25T19:09:20.511 INFO:teuthology.orchestra.run.smithi028.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 764 kB/s | 32 kB 00:00 2024-04-25T19:09:20.556 INFO:teuthology.orchestra.run.smithi053.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-04-25T19:09:20.586 INFO:teuthology.orchestra.run.smithi028.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 9.9 MB/s | 756 kB 00:00 2024-04-25T19:09:20.589 INFO:teuthology.orchestra.run.smithi053.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-04-25T19:09:20.637 INFO:teuthology.orchestra.run.smithi028.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 3.1 MB/s | 158 kB 00:00 2024-04-25T19:09:20.673 INFO:teuthology.orchestra.run.smithi053.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-25T19:09:20.687 INFO:teuthology.orchestra.run.smithi028.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 2.5 MB/s | 127 kB 00:00 2024-04-25T19:09:20.698 INFO:teuthology.orchestra.run.smithi053.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 986 kB/s | 106 kB 00:00 2024-04-25T19:09:20.732 INFO:teuthology.orchestra.run.smithi053.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-04-25T19:09:20.746 INFO:teuthology.orchestra.run.smithi028.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 8.4 MB/s | 497 kB 00:00 2024-04-25T19:09:20.807 INFO:teuthology.orchestra.run.smithi053.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-04-25T19:09:20.832 INFO:teuthology.orchestra.run.smithi053.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 934 kB/s | 125 kB 00:00 2024-04-25T19:09:20.883 INFO:teuthology.orchestra.run.smithi053.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 9.0 MB/s | 689 kB 00:00 2024-04-25T19:09:20.916 INFO:teuthology.orchestra.run.smithi053.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 943 kB 00:00 2024-04-25T19:09:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T19:09:20.958 INFO:teuthology.orchestra.run.smithi053.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 519 kB/s | 21 kB 00:00 2024-04-25T19:09:20.983 INFO:teuthology.orchestra.run.smithi053.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 313 kB 00:00 2024-04-25T19:09:21.001 INFO:teuthology.orchestra.run.smithi053.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 788 kB/s | 33 kB 00:00 2024-04-25T19:09:21.054 INFO:teuthology.orchestra.run.smithi028.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 17 MB/s | 5.2 MB 00:00 2024-04-25T19:09:21.059 INFO:teuthology.orchestra.run.smithi053.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-04-25T19:09:21.096 INFO:teuthology.orchestra.run.smithi028.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 1.1 MB/s | 45 kB 00:00 2024-04-25T19:09:21.146 INFO:teuthology.orchestra.run.smithi028.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 2.7 MB/s | 135 kB 00:00 2024-04-25T19:09:21.205 INFO:teuthology.orchestra.run.smithi028.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.7 MB/s | 161 kB 00:00 2024-04-25T19:09:21.218 INFO:teuthology.orchestra.run.smithi053.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2024-04-25T19:09:21.243 INFO:teuthology.orchestra.run.smithi053.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 61 kB/s | 11 kB 00:00 2024-04-25T19:09:21.264 INFO:teuthology.orchestra.run.smithi028.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 5.4 MB/s | 321 kB 00:00 2024-04-25T19:09:21.293 INFO:teuthology.orchestra.run.smithi053.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 910 kB/s | 45 kB 00:00 2024-04-25T19:09:21.318 INFO:teuthology.orchestra.run.smithi053.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-25T19:09:21.322 INFO:teuthology.orchestra.run.smithi028.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 5.0 MB/s | 298 kB 00:00 2024-04-25T19:09:21.369 INFO:teuthology.orchestra.run.smithi053.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 8.7 MB/s | 662 kB 00:00 2024-04-25T19:09:21.373 INFO:teuthology.orchestra.run.smithi028.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.9 MB/s | 100 kB 00:00 2024-04-25T19:09:21.394 INFO:teuthology.orchestra.run.smithi053.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 582 kB 00:00 2024-04-25T19:09:21.423 INFO:teuthology.orchestra.run.smithi028.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.7 MB/s | 86 kB 00:00 2024-04-25T19:09:21.436 INFO:teuthology.orchestra.run.smithi053.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 388 kB/s | 25 kB 00:00 2024-04-25T19:09:21.461 INFO:teuthology.orchestra.run.smithi053.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 709 kB/s | 47 kB 00:00 2024-04-25T19:09:21.503 INFO:teuthology.orchestra.run.smithi053.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-04-25T19:09:21.528 INFO:teuthology.orchestra.run.smithi053.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-25T19:09:21.564 INFO:teuthology.orchestra.run.smithi053.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 580 kB/s | 35 kB 00:00 2024-04-25T19:09:21.615 INFO:teuthology.orchestra.run.smithi028.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 16 MB/s | 3.0 MB 00:00 2024-04-25T19:09:21.665 INFO:teuthology.orchestra.run.smithi028.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 3.3 MB/s | 169 kB 00:00 2024-04-25T19:09:21.708 INFO:teuthology.orchestra.run.smithi028.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 817 kB/s | 34 kB 00:00 2024-04-25T19:09:21.758 INFO:teuthology.orchestra.run.smithi028.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.8 MB/s | 142 kB 00:00 2024-04-25T19:09:21.773 INFO:teuthology.orchestra.run.smithi053.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-04-25T19:09:21.840 INFO:teuthology.orchestra.run.smithi053.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-25T19:09:21.882 INFO:teuthology.orchestra.run.smithi053.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-25T19:09:21.924 INFO:teuthology.orchestra.run.smithi053.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-25T19:09:21.949 INFO:teuthology.orchestra.run.smithi053.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-25T19:09:22.000 INFO:teuthology.orchestra.run.smithi028.stdout:(30/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 2.9 MB/s | 18 MB 00:06 2024-04-25T19:09:22.032 INFO:teuthology.orchestra.run.smithi053.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-25T19:09:22.067 INFO:teuthology.orchestra.run.smithi028.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 11 MB/s | 3.5 MB 00:00 2024-04-25T19:09:22.118 INFO:teuthology.orchestra.run.smithi028.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 4.9 MB/s | 249 kB 00:00 2024-04-25T19:09:22.133 INFO:teuthology.orchestra.run.smithi053.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 28 MB/s | 5.8 MB 00:00 2024-04-25T19:09:22.158 INFO:teuthology.orchestra.run.smithi053.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 713 kB/s | 87 kB 00:00 2024-04-25T19:09:22.159 INFO:teuthology.orchestra.run.smithi028.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 1.2 MB/s | 49 kB 00:00 2024-04-25T19:09:22.201 INFO:teuthology.orchestra.run.smithi028.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 397 kB/s | 16 kB 00:00 2024-04-25T19:09:22.208 INFO:teuthology.orchestra.run.smithi053.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.9 MB/s | 244 kB 00:00 2024-04-25T19:09:22.251 INFO:teuthology.orchestra.run.smithi028.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 5.3 MB/s | 273 kB 00:00 2024-04-25T19:09:22.275 INFO:teuthology.orchestra.run.smithi053.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.8 MB/s | 248 kB 00:00 2024-04-25T19:09:22.335 INFO:teuthology.orchestra.run.smithi028.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 9.0 MB/s | 762 kB 00:00 2024-04-25T19:09:22.351 INFO:teuthology.orchestra.run.smithi053.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 6.0 MB 00:00 2024-04-25T19:09:22.367 INFO:teuthology.orchestra.run.smithi053.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 282 kB/s | 26 kB 00:00 2024-04-25T19:09:22.377 INFO:teuthology.orchestra.run.smithi028.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 439 kB/s | 18 kB 00:00 2024-04-25T19:09:22.384 INFO:teuthology.orchestra.run.smithi053.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 572 kB/s | 19 kB 00:00 2024-04-25T19:09:22.426 INFO:teuthology.orchestra.run.smithi053.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-25T19:09:22.443 INFO:teuthology.orchestra.run.smithi053.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 594 kB/s | 35 kB 00:00 2024-04-25T19:09:22.494 INFO:teuthology.orchestra.run.smithi053.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-04-25T19:09:22.577 INFO:teuthology.orchestra.run.smithi028.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 401 kB/s | 80 kB 00:00 2024-04-25T19:09:22.602 INFO:teuthology.orchestra.run.smithi053.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-04-25T19:09:22.627 INFO:teuthology.orchestra.run.smithi053.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-04-25T19:09:22.653 INFO:teuthology.orchestra.run.smithi053.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T19:09:22.678 INFO:teuthology.orchestra.run.smithi053.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 839 kB/s | 42 kB 00:00 2024-04-25T19:09:22.686 INFO:teuthology.orchestra.run.smithi028.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 7.3 MB/s | 803 kB 00:00 2024-04-25T19:09:22.703 INFO:teuthology.orchestra.run.smithi053.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-25T19:09:22.728 INFO:teuthology.orchestra.run.smithi028.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-04-25T19:09:22.728 INFO:teuthology.orchestra.run.smithi053.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T19:09:22.754 INFO:teuthology.orchestra.run.smithi053.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-25T19:09:22.770 INFO:teuthology.orchestra.run.smithi028.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.9 MB/s | 253 kB 00:00 2024-04-25T19:09:22.779 INFO:teuthology.orchestra.run.smithi053.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T19:09:22.804 INFO:teuthology.orchestra.run.smithi053.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T19:09:22.870 INFO:teuthology.orchestra.run.smithi028.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-04-25T19:09:22.871 INFO:teuthology.orchestra.run.smithi053.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 625 kB/s | 42 kB 00:00 2024-04-25T19:09:22.929 INFO:teuthology.orchestra.run.smithi028.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-25T19:09:22.955 INFO:teuthology.orchestra.run.smithi053.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 548 kB/s | 44 kB 00:00 2024-04-25T19:09:23.004 INFO:teuthology.orchestra.run.smithi028.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 135 kB 00:00 2024-04-25T19:09:23.105 INFO:teuthology.orchestra.run.smithi053.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.5 MB/s | 221 kB 00:00 2024-04-25T19:09:23.121 INFO:teuthology.orchestra.run.smithi028.stdout:(45/152): ceph-mgr-diskprediction-local-19.0.0- 6.6 MB/s | 7.4 MB 00:01 2024-04-25T19:09:23.146 INFO:teuthology.orchestra.run.smithi028.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 880 kB/s | 125 kB 00:00 2024-04-25T19:09:23.188 INFO:teuthology.orchestra.run.smithi028.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-04-25T19:09:23.239 INFO:teuthology.orchestra.run.smithi028.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 7.3 MB/s | 689 kB 00:00 2024-04-25T19:09:23.272 INFO:teuthology.orchestra.run.smithi028.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 11 MB/s | 943 kB 00:00 2024-04-25T19:09:23.298 INFO:teuthology.orchestra.run.smithi028.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T19:09:23.323 INFO:teuthology.orchestra.run.smithi028.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-25T19:09:23.348 INFO:teuthology.orchestra.run.smithi053.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-25T19:09:23.356 INFO:teuthology.orchestra.run.smithi028.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-25T19:09:23.373 INFO:teuthology.orchestra.run.smithi053.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 136 kB/s | 36 kB 00:00 2024-04-25T19:09:23.382 INFO:teuthology.orchestra.run.smithi028.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 557 kB/s | 33 kB 00:00 2024-04-25T19:09:23.398 INFO:teuthology.orchestra.run.smithi053.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-25T19:09:23.449 INFO:teuthology.orchestra.run.smithi053.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 305 kB 00:00 2024-04-25T19:09:23.457 INFO:teuthology.orchestra.run.smithi028.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 243 kB/s | 18 kB 00:00 2024-04-25T19:09:23.474 INFO:teuthology.orchestra.run.smithi053.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 768 kB/s | 58 kB 00:00 2024-04-25T19:09:23.516 INFO:teuthology.orchestra.run.smithi053.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.4 MB/s | 94 kB 00:00 2024-04-25T19:09:23.541 INFO:teuthology.orchestra.run.smithi053.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 951 kB/s | 64 kB 00:00 2024-04-25T19:09:23.574 INFO:teuthology.orchestra.run.smithi028.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-25T19:09:23.575 INFO:teuthology.orchestra.run.smithi053.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 765 kB/s | 45 kB 00:00 2024-04-25T19:09:23.599 INFO:teuthology.orchestra.run.smithi028.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 79 kB/s | 11 kB 00:00 2024-04-25T19:09:23.600 INFO:teuthology.orchestra.run.smithi053.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 380 kB/s | 22 kB 00:00 2024-04-25T19:09:23.641 INFO:teuthology.orchestra.run.smithi028.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 160 kB 00:00 2024-04-25T19:09:23.675 INFO:teuthology.orchestra.run.smithi028.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 602 kB/s | 45 kB 00:00 2024-04-25T19:09:23.675 INFO:teuthology.orchestra.run.smithi053.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 8.4 MB/s | 863 kB 00:00 2024-04-25T19:09:23.725 INFO:teuthology.orchestra.run.smithi028.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-04-25T19:09:23.767 INFO:teuthology.orchestra.run.smithi028.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-04-25T19:09:23.792 INFO:teuthology.orchestra.run.smithi028.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 383 kB/s | 25 kB 00:00 2024-04-25T19:09:23.826 INFO:teuthology.orchestra.run.smithi028.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-04-25T19:09:23.876 INFO:teuthology.orchestra.run.smithi028.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-04-25T19:09:23.902 INFO:teuthology.orchestra.run.smithi028.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-25T19:09:23.943 INFO:teuthology.orchestra.run.smithi028.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 35 kB 00:00 2024-04-25T19:09:24.342 INFO:teuthology.orchestra.run.smithi053.stdout:(104/152): ceph-mgr-diskprediction-local-19.0.0 1.5 MB/s | 7.4 MB 00:05 2024-04-25T19:09:24.367 INFO:teuthology.orchestra.run.smithi053.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 446 kB/s | 309 kB 00:00 2024-04-25T19:09:24.385 INFO:teuthology.orchestra.run.smithi028.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-25T19:09:24.401 INFO:teuthology.orchestra.run.smithi053.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 684 kB/s | 548 kB 00:00 2024-04-25T19:09:24.461 INFO:teuthology.orchestra.run.smithi028.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 8.5 MB/s | 4.7 MB 00:00 2024-04-25T19:09:24.501 INFO:teuthology.orchestra.run.smithi053.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-04-25T19:09:24.553 INFO:teuthology.orchestra.run.smithi028.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 9.7 MB/s | 1.6 MB 00:00 2024-04-25T19:09:24.601 INFO:teuthology.orchestra.run.smithi053.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 487 kB/s | 49 kB 00:00 2024-04-25T19:09:24.602 INFO:teuthology.orchestra.run.smithi028.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 6.4 MB/s | 911 kB 00:00 2024-04-25T19:09:24.628 INFO:teuthology.orchestra.run.smithi028.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.8 MB/s | 133 kB 00:00 2024-04-25T19:09:24.693 INFO:teuthology.orchestra.run.smithi053.stdout:(109/152): grpc-data-1.46.7-10.el9.noarch.rpm 55 kB/s | 19 kB 00:00 2024-04-25T19:09:24.719 INFO:teuthology.orchestra.run.smithi053.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 576 kB/s | 67 kB 00:00 2024-04-25T19:09:24.837 INFO:teuthology.orchestra.run.smithi028.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-04-25T19:09:24.869 INFO:teuthology.orchestra.run.smithi053.stdout:(111/152): libarrow-9.0.0-10.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-04-25T19:09:24.911 INFO:teuthology.orchestra.run.smithi053.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-04-25T19:09:24.986 INFO:teuthology.orchestra.run.smithi053.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-25T19:09:25.012 INFO:teuthology.orchestra.run.smithi053.stdout:(114/152): luarocks-3.9.2-1.el9.noarch.rpm 476 kB/s | 151 kB 00:00 2024-04-25T19:09:25.028 INFO:teuthology.orchestra.run.smithi053.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 371 kB/s | 43 kB 00:00 2024-04-25T19:09:25.029 INFO:teuthology.orchestra.run.smithi028.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 14 MB/s | 5.8 MB 00:00 2024-04-25T19:09:25.054 INFO:teuthology.orchestra.run.smithi028.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 407 kB/s | 87 kB 00:00 2024-04-25T19:09:25.095 INFO:teuthology.orchestra.run.smithi053.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 296 kB/s | 32 kB 00:00 2024-04-25T19:09:25.154 INFO:teuthology.orchestra.run.smithi028.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.4 MB/s | 244 kB 00:00 2024-04-25T19:09:25.162 INFO:teuthology.orchestra.run.smithi053.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 99 kB/s | 15 kB 00:00 2024-04-25T19:09:25.204 INFO:teuthology.orchestra.run.smithi053.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 979 kB/s | 172 kB 00:00 2024-04-25T19:09:25.229 INFO:teuthology.orchestra.run.smithi053.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2024-04-25T19:09:25.263 INFO:teuthology.orchestra.run.smithi053.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-25T19:09:25.305 INFO:teuthology.orchestra.run.smithi053.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-25T19:09:25.330 INFO:teuthology.orchestra.run.smithi053.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-25T19:09:25.414 INFO:teuthology.orchestra.run.smithi053.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-04-25T19:09:25.429 INFO:teuthology.orchestra.run.smithi028.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-25T19:09:25.431 INFO:teuthology.orchestra.run.smithi053.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-04-25T19:09:25.455 INFO:teuthology.orchestra.run.smithi028.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 828 kB/s | 248 kB 00:00 2024-04-25T19:09:25.456 INFO:teuthology.orchestra.run.smithi053.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 958 kB/s | 144 kB 00:00 2024-04-25T19:09:25.497 INFO:teuthology.orchestra.run.smithi028.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 385 kB/s | 26 kB 00:00 2024-04-25T19:09:25.515 INFO:teuthology.orchestra.run.smithi053.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 175 kB/s | 18 kB 00:00 2024-04-25T19:09:25.522 INFO:teuthology.orchestra.run.smithi028.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 285 kB/s | 19 kB 00:00 2024-04-25T19:09:25.540 INFO:teuthology.orchestra.run.smithi053.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-04-25T19:09:25.565 INFO:teuthology.orchestra.run.smithi053.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-04-25T19:09:25.572 INFO:teuthology.orchestra.run.smithi028.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T19:09:25.598 INFO:teuthology.orchestra.run.smithi028.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 461 kB/s | 35 kB 00:00 2024-04-25T19:09:25.616 INFO:teuthology.orchestra.run.smithi053.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-25T19:09:25.666 INFO:teuthology.orchestra.run.smithi053.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 8.1 MB/s | 1.0 MB 00:00 2024-04-25T19:09:25.691 INFO:teuthology.orchestra.run.smithi053.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 369 kB/s | 46 kB 00:00 2024-04-25T19:09:25.725 INFO:teuthology.orchestra.run.smithi053.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 724 kB/s | 79 kB 00:00 2024-04-25T19:09:25.767 INFO:teuthology.orchestra.run.smithi053.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-04-25T19:09:25.773 INFO:teuthology.orchestra.run.smithi028.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 250 kB 00:00 2024-04-25T19:09:25.825 INFO:teuthology.orchestra.run.smithi053.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-25T19:09:25.859 INFO:teuthology.orchestra.run.smithi053.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-04-25T19:09:25.884 INFO:teuthology.orchestra.run.smithi053.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 768 kB/s | 90 kB 00:00 2024-04-25T19:09:25.926 INFO:teuthology.orchestra.run.smithi053.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-04-25T19:09:25.960 INFO:teuthology.orchestra.run.smithi053.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-04-25T19:09:25.985 INFO:teuthology.orchestra.run.smithi053.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-25T19:09:26.027 INFO:teuthology.orchestra.run.smithi053.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-04-25T19:09:26.031 INFO:teuthology.orchestra.run.smithi028.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-25T19:09:26.057 INFO:teuthology.orchestra.run.smithi028.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 787 kB/s | 222 kB 00:00 2024-04-25T19:09:26.060 INFO:teuthology.orchestra.run.smithi053.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-04-25T19:09:26.082 INFO:teuthology.orchestra.run.smithi028.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T19:09:26.086 INFO:teuthology.orchestra.run.smithi053.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 506 kB/s | 51 kB 00:00 2024-04-25T19:09:26.136 INFO:teuthology.orchestra.run.smithi053.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-25T19:09:26.186 INFO:teuthology.orchestra.run.smithi053.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 714 kB/s | 90 kB 00:00 2024-04-25T19:09:26.190 INFO:teuthology.orchestra.run.smithi028.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 319 kB/s | 42 kB 00:00 2024-04-25T19:09:26.211 INFO:teuthology.orchestra.run.smithi053.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-25T19:09:26.216 INFO:teuthology.orchestra.run.smithi028.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 269 kB 00:00 2024-04-25T19:09:26.241 INFO:teuthology.orchestra.run.smithi028.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T19:09:26.245 INFO:teuthology.orchestra.run.smithi053.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 183 kB/s | 20 kB 00:00 2024-04-25T19:09:26.275 INFO:teuthology.orchestra.run.smithi028.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-04-25T19:09:26.301 INFO:teuthology.orchestra.run.smithi028.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 874 kB/s | 51 kB 00:00 2024-04-25T19:09:26.326 INFO:teuthology.orchestra.run.smithi028.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-25T19:09:26.328 INFO:teuthology.orchestra.run.smithi053.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-04-25T19:09:26.371 INFO:teuthology.orchestra.run.smithi053.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-25T19:09:26.426 INFO:teuthology.orchestra.run.smithi028.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 418 kB/s | 42 kB 00:00 2024-04-25T19:09:26.462 INFO:teuthology.orchestra.run.smithi053.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 15 MB/s | 1.3 MB 00:00 2024-04-25T19:09:26.596 INFO:teuthology.orchestra.run.smithi053.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 18 MB/s | 2.4 MB 00:00 2024-04-25T19:09:26.701 INFO:teuthology.orchestra.run.smithi028.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 161 kB/s | 44 kB 00:00 2024-04-25T19:09:26.943 INFO:teuthology.orchestra.run.smithi028.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 921 kB/s | 221 kB 00:00 2024-04-25T19:09:27.094 INFO:teuthology.orchestra.run.smithi028.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-04-25T19:09:27.119 INFO:teuthology.orchestra.run.smithi028.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 207 kB/s | 36 kB 00:00 2024-04-25T19:09:27.144 INFO:teuthology.orchestra.run.smithi028.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 951 kB/s | 47 kB 00:00 2024-04-25T19:09:27.169 INFO:teuthology.orchestra.run.smithi028.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-04-25T19:09:27.195 INFO:teuthology.orchestra.run.smithi028.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T19:09:27.220 INFO:teuthology.orchestra.run.smithi028.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T19:09:27.237 INFO:teuthology.orchestra.run.smithi028.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-25T19:09:27.254 INFO:teuthology.orchestra.run.smithi028.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-25T19:09:27.271 INFO:teuthology.orchestra.run.smithi028.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 670 kB/s | 22 kB 00:00 2024-04-25T19:09:27.329 INFO:teuthology.orchestra.run.smithi028.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 11 MB/s | 863 kB 00:00 2024-04-25T19:09:27.471 INFO:teuthology.orchestra.run.smithi053.stdout:(151/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 2.7 MB/s | 3.1 MB 00:01 2024-04-25T19:09:27.646 INFO:teuthology.orchestra.run.smithi053.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 2.4 MB/s | 3.3 MB 00:01 2024-04-25T19:09:27.654 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:09:27.654 INFO:teuthology.orchestra.run.smithi053.stdout:Total 13 MB/s | 214 MB 00:16 2024-04-25T19:09:27.821 INFO:teuthology.orchestra.run.smithi028.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 628 kB/s | 309 kB 00:00 2024-04-25T19:09:27.863 INFO:teuthology.orchestra.run.smithi028.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 926 kB/s | 548 kB 00:00 2024-04-25T19:09:27.922 INFO:teuthology.orchestra.run.smithi028.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 193 kB/s | 19 kB 00:00 2024-04-25T19:09:28.030 INFO:teuthology.orchestra.run.smithi028.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-04-25T19:09:28.139 INFO:teuthology.orchestra.run.smithi028.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-04-25T19:09:28.248 INFO:teuthology.orchestra.run.smithi028.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-04-25T19:09:28.432 INFO:teuthology.orchestra.run.smithi028.stdout:(110/152): ceph-common-19.0.0-3291.g994e2f20.el 1.3 MB/s | 20 MB 00:14 2024-04-25T19:09:28.498 INFO:teuthology.orchestra.run.smithi028.stdout:(111/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-04-25T19:09:28.524 INFO:teuthology.orchestra.run.smithi028.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 548 kB/s | 151 kB 00:00 2024-04-25T19:09:28.615 INFO:teuthology.orchestra.run.smithi028.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-04-25T19:09:28.641 INFO:teuthology.orchestra.run.smithi028.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 371 kB/s | 43 kB 00:00 2024-04-25T19:09:28.716 INFO:teuthology.orchestra.run.smithi028.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-04-25T19:09:28.741 INFO:teuthology.orchestra.run.smithi028.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-25T19:09:28.788 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-04-25T19:09:28.825 INFO:teuthology.orchestra.run.smithi028.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-25T19:09:28.925 INFO:teuthology.orchestra.run.smithi028.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2024-04-25T19:09:28.950 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-04-25T19:09:28.950 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-04-25T19:09:28.950 INFO:teuthology.orchestra.run.smithi028.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-04-25T19:09:29.026 INFO:teuthology.orchestra.run.smithi028.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 353 kB/s | 35 kB 00:00 2024-04-25T19:09:29.051 INFO:teuthology.orchestra.run.smithi028.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-25T19:09:29.151 INFO:teuthology.orchestra.run.smithi028.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-25T19:09:29.243 INFO:teuthology.orchestra.run.smithi028.stdout:(123/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.0 MB/s | 838 kB 00:00 2024-04-25T19:09:29.268 INFO:teuthology.orchestra.run.smithi028.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-04-25T19:09:29.335 INFO:teuthology.orchestra.run.smithi028.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-04-25T19:09:29.369 INFO:teuthology.orchestra.run.smithi028.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-25T19:09:29.428 INFO:teuthology.orchestra.run.smithi028.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-25T19:09:29.470 INFO:teuthology.orchestra.run.smithi028.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-04-25T19:09:29.545 INFO:teuthology.orchestra.run.smithi028.stdout:(129/152): python3-kubernetes-26.1.0-2.el9.noar 8.7 MB/s | 1.0 MB 00:00 2024-04-25T19:09:29.587 INFO:teuthology.orchestra.run.smithi028.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 397 kB/s | 46 kB 00:00 2024-04-25T19:09:29.629 INFO:teuthology.orchestra.run.smithi028.stdout:(131/152): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-04-25T19:09:29.654 INFO:teuthology.orchestra.run.smithi028.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 725 kB/s | 79 kB 00:00 2024-04-25T19:09:29.679 INFO:teuthology.orchestra.run.smithi028.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 625 kB/s | 58 kB 00:00 2024-04-25T19:09:29.730 INFO:teuthology.orchestra.run.smithi028.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-25T19:09:29.755 INFO:teuthology.orchestra.run.smithi028.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-25T19:09:29.780 INFO:teuthology.orchestra.run.smithi028.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 893 kB/s | 90 kB 00:00 2024-04-25T19:09:29.822 INFO:teuthology.orchestra.run.smithi028.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 305 kB/s | 28 kB 00:00 2024-04-25T19:09:29.856 INFO:teuthology.orchestra.run.smithi028.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 305 kB/s | 31 kB 00:00 2024-04-25T19:09:29.914 INFO:teuthology.orchestra.run.smithi028.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-04-25T19:09:29.948 INFO:teuthology.orchestra.run.smithi028.stdout:(140/152): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-04-25T19:09:29.965 INFO:teuthology.orchestra.run.smithi028.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-04-25T19:09:30.024 INFO:teuthology.orchestra.run.smithi028.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-04-25T19:09:30.166 INFO:teuthology.orchestra.run.smithi028.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-04-25T19:09:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 398 kB/s | 90 kB 00:00 2024-04-25T19:09:30.233 INFO:teuthology.orchestra.run.smithi028.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.0 MB/s | 427 kB 00:00 2024-04-25T19:09:30.258 INFO:teuthology.orchestra.run.smithi028.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-04-25T19:09:30.367 INFO:teuthology.orchestra.run.smithi028.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-04-25T19:09:30.575 INFO:teuthology.orchestra.run.smithi028.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-04-25T19:09:30.651 INFO:teuthology.orchestra.run.smithi028.stdout:(149/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 11 MB/s | 3.1 MB 00:00 2024-04-25T19:09:30.692 INFO:teuthology.orchestra.run.smithi028.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 12 MB/s | 1.3 MB 00:00 2024-04-25T19:09:30.774 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-04-25T19:09:30.775 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-04-25T19:09:30.784 INFO:teuthology.orchestra.run.smithi028.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 18 MB/s | 2.4 MB 00:00 2024-04-25T19:09:30.843 INFO:teuthology.orchestra.run.smithi028.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 5.7 MB/s | 3.3 MB 00:00 2024-04-25T19:09:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-25T19:09:30.847 INFO:teuthology.orchestra.run.smithi028.stdout:Total 12 MB/s | 214 MB 00:17 2024-04-25T19:09:31.991 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-25T19:09:32.151 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-25T19:09:32.151 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-25T19:09:33.994 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-25T19:09:33.994 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-25T19:09:36.765 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-04-25T19:09:36.859 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T19:09:36.898 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T19:09:36.960 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T19:09:37.002 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T19:09:37.027 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T19:09:37.070 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T19:09:37.423 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T19:09:37.536 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T19:09:37.664 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T19:09:37.695 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T19:09:37.788 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T19:09:37.817 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T19:09:37.979 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T19:09:38.031 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T19:09:38.058 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T19:09:38.164 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T19:09:38.192 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T19:09:38.247 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T19:09:38.268 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T19:09:38.290 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T19:09:38.528 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T19:09:38.560 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T19:09:38.673 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T19:09:38.769 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T19:09:38.800 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T19:09:38.836 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T19:09:38.912 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T19:09:38.952 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T19:09:38.995 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T19:09:39.028 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T19:09:39.055 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T19:09:39.120 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T19:09:39.171 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T19:09:39.200 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T19:09:39.244 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T19:09:39.323 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T19:09:39.355 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T19:09:39.399 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T19:09:39.427 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T19:09:39.502 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T19:09:39.559 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T19:09:39.637 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T19:09:39.785 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T19:09:39.846 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T19:09:40.143 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-25T19:09:40.238 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T19:09:40.278 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T19:09:40.332 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T19:09:40.384 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T19:09:40.409 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T19:09:40.418 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T19:09:40.453 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T19:09:40.502 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T19:09:40.535 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T19:09:40.579 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T19:09:40.617 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T19:09:40.659 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T19:09:40.827 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T19:09:40.868 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T19:09:40.878 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T19:09:40.926 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T19:09:40.935 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T19:09:41.010 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T19:09:41.045 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T19:09:41.054 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T19:09:41.081 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T19:09:41.094 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T19:09:41.130 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T19:09:41.165 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T19:09:41.169 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T19:09:41.206 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T19:09:41.208 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T19:09:41.251 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T19:09:41.366 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T19:09:41.397 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T19:09:41.429 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T19:09:41.439 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T19:09:41.456 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T19:09:41.468 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T19:09:41.511 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T19:09:41.563 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T19:09:41.591 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T19:09:41.611 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T19:09:41.651 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T19:09:41.653 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T19:09:41.675 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T19:09:41.692 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T19:09:41.697 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T19:09:41.742 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T19:09:41.767 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T19:09:41.848 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T19:09:41.902 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T19:09:41.933 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T19:09:41.977 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T19:09:42.008 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T19:09:42.044 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T19:09:42.141 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T19:09:42.174 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T19:09:42.203 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T19:09:42.209 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T19:09:42.294 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T19:09:42.334 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T19:09:42.377 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T19:09:42.396 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T19:09:42.409 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T19:09:42.437 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T19:09:42.510 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T19:09:42.554 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T19:09:42.561 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T19:09:42.590 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T19:09:42.593 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T19:09:42.621 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T19:09:42.634 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T19:09:42.656 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T19:09:42.689 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T19:09:42.712 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T19:09:42.732 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T19:09:42.744 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T19:09:42.765 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T19:09:42.789 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T19:09:42.817 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T19:09:42.859 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T19:09:42.889 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T19:09:42.913 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T19:09:42.932 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T19:09:42.953 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T19:09:43.000 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T19:09:43.009 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T19:09:43.112 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T19:09:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T19:09:43.162 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T19:09:43.219 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T19:09:43.787 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T19:09:43.875 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T19:09:43.908 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T19:09:43.948 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T19:09:43.951 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T19:09:43.989 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T19:09:44.000 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T19:09:44.032 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T19:09:44.033 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T19:09:44.139 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T19:09:44.192 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T19:09:44.202 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T19:09:44.251 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T19:09:44.284 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T19:09:44.307 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T19:09:44.318 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T19:09:44.381 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T19:09:44.419 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T19:09:44.455 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T19:09:44.502 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T19:09:44.503 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T19:09:44.537 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T19:09:44.541 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T19:09:44.614 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T19:09:44.621 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T19:09:44.656 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T19:09:44.799 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T19:09:44.845 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T19:09:44.875 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T19:09:44.917 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T19:09:45.015 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T19:09:45.058 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T19:09:45.098 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T19:09:45.123 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T19:09:45.156 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T19:09:45.235 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T19:09:45.318 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T19:09:45.364 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T19:09:45.397 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T19:09:45.425 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T19:09:45.444 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T19:09:45.459 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T19:09:45.499 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T19:09:45.538 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T19:09:45.647 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T19:09:45.686 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T19:09:45.719 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T19:09:45.762 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T19:09:45.796 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T19:09:45.887 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T19:09:45.935 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T19:09:45.965 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T19:09:46.005 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T19:09:46.118 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T19:09:46.169 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T19:09:46.767 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T19:09:46.854 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T19:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T19:09:46.915 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T19:09:46.955 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T19:09:47.006 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T19:09:47.048 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T19:09:47.168 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T19:09:47.252 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T19:09:47.302 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T19:09:47.349 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T19:09:47.394 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T19:09:47.424 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T19:09:47.441 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T19:09:47.460 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T19:09:47.475 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T19:09:47.660 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T19:09:47.689 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T19:09:47.769 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T19:09:47.977 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T19:09:48.005 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T19:09:48.076 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T19:09:48.107 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T19:09:48.465 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T19:09:48.585 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T19:09:49.917 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T19:09:49.994 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T19:09:50.013 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T19:09:50.046 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T19:09:50.385 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T19:09:50.426 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T19:09:50.440 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T19:09:50.478 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T19:09:50.517 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T19:09:50.547 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T19:09:50.583 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T19:09:51.069 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T19:09:51.100 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T19:09:51.115 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T19:09:51.137 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T19:09:51.191 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T19:09:51.211 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T19:09:51.211 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T19:09:51.211 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T19:09:51.211 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:09:51.230 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T19:09:51.241 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T19:09:53.579 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T19:09:53.618 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T19:09:54.205 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T19:09:54.255 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T19:09:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T19:09:54.347 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T19:09:54.347 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T19:09:54.347 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:09:54.373 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T19:10:03.337 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T19:10:03.337 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-04-25T19:10:03.337 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-04-25T19:10:03.338 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-04-25T19:10:03.338 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-04-25T19:10:03.338 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-04-25T19:10:03.338 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-04-25T19:10:03.338 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-04-25T19:10:03.338 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:03.512 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T19:10:03.528 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T19:10:03.528 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:03.529 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T19:10:03.529 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:05.165 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T19:10:05.193 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T19:10:05.193 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:05.193 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T19:10:05.193 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T19:10:05.193 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T19:10:05.193 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:05.424 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T19:10:05.448 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T19:10:05.448 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:05.448 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T19:10:05.448 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T19:10:05.448 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T19:10:05.448 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:05.933 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T19:10:05.956 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T19:10:05.956 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:05.957 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T19:10:05.957 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T19:10:05.957 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T19:10:05.957 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:06.072 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T19:10:06.131 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T19:10:06.195 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T19:10:06.393 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T19:10:06.451 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-04-25T19:10:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:06.534 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T19:10:06.632 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T19:10:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T19:10:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T19:10:06.677 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:06.677 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T19:10:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:06.776 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T19:10:06.818 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T19:10:06.850 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T19:10:06.877 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T19:10:07.313 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T19:10:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T19:10:07.383 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T19:10:07.447 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T19:10:07.482 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T19:10:08.313 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T19:10:08.341 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T19:10:08.342 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:08.342 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T19:10:08.342 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T19:10:08.342 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T19:10:08.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:08.581 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T19:10:08.587 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T19:10:08.605 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T19:10:08.605 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:08.605 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T19:10:08.605 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T19:10:08.605 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T19:10:08.606 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:08.672 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T19:10:09.090 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T19:10:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T19:10:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T19:10:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T19:10:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T19:10:09.114 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:09.229 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T19:10:09.279 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T19:10:09.360 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T19:10:09.557 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T19:10:09.600 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T19:10:09.681 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T19:10:09.694 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T19:10:09.725 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T19:10:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T19:10:09.836 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T19:10:09.923 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T19:10:09.966 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T19:10:09.998 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T19:10:10.008 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T19:10:10.025 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T19:10:10.036 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T19:10:10.060 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T19:10:10.060 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:10.060 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T19:10:10.060 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T19:10:10.060 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T19:10:10.060 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:10.111 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T19:10:10.123 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T19:10:10.226 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T19:10:10.279 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T19:10:10.479 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T19:10:10.507 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T19:10:10.540 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T19:10:10.605 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T19:10:10.630 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T19:10:11.266 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T19:10:11.297 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T19:10:11.321 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T19:10:11.321 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:11.321 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T19:10:11.321 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T19:10:11.321 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T19:10:11.322 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:11.365 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T19:10:11.388 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T19:10:11.388 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:11.388 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T19:10:11.388 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:11.440 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T19:10:11.464 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T19:10:11.464 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:11.464 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T19:10:11.464 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T19:10:11.464 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T19:10:11.465 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:11.735 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T19:10:11.774 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T19:10:11.816 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T19:10:11.840 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T19:10:11.840 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:11.840 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T19:10:11.841 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T19:10:11.841 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T19:10:11.841 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:12.792 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T19:10:12.824 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T19:10:12.934 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T19:10:13.107 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T19:10:13.135 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T19:10:13.159 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T19:10:13.159 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:13.159 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T19:10:13.160 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T19:10:13.160 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T19:10:13.160 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:13.210 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T19:10:13.222 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T19:10:13.324 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T19:10:13.376 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T19:10:14.364 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T19:10:14.404 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T19:10:14.428 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T19:10:14.428 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:14.428 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T19:10:14.428 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T19:10:14.428 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T19:10:14.429 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:14.472 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T19:10:14.495 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T19:10:14.495 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:14.495 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T19:10:14.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:14.547 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T19:10:14.571 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T19:10:14.571 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:14.571 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T19:10:14.571 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T19:10:14.571 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T19:10:14.572 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:14.898 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T19:10:14.922 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T19:10:14.922 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T19:10:14.922 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T19:10:14.922 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T19:10:14.923 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T19:10:14.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:16.362 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T19:10:16.404 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T19:10:16.445 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T19:10:16.476 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T19:10:16.503 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T19:10:16.710 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T19:10:16.758 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T19:10:16.787 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T19:10:17.432 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T19:10:17.476 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T19:10:17.497 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T19:10:18.435 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T19:10:18.523 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T19:10:18.565 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T19:10:18.661 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T19:10:18.702 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T19:10:18.702 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T19:10:18.795 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T19:10:18.827 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T19:10:18.827 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T19:10:18.920 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T19:10:18.987 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T19:10:19.453 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T19:10:19.494 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T19:10:19.536 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T19:10:19.566 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T19:10:19.594 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T19:10:19.798 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T19:10:19.840 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T19:10:19.869 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T19:10:20.497 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T19:10:20.534 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T19:10:20.555 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T19:10:21.473 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T19:10:21.572 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T19:10:21.614 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T19:10:21.709 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T19:10:21.743 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T19:10:21.743 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T19:10:21.843 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T19:10:21.876 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T19:10:21.877 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T19:10:21.965 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T19:10:22.046 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T19:10:24.797 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T19:10:24.798 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T19:10:24.799 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T19:10:24.800 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T19:10:24.801 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T19:10:24.802 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T19:10:24.803 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T19:10:24.804 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T19:10:24.805 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T19:10:24.806 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T19:10:24.807 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T19:10:24.808 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T19:10:24.809 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T19:10:24.810 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T19:10:24.811 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T19:10:26.956 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T19:10:26.957 INFO:teuthology.orchestra.run.smithi053.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.958 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T19:10:26.960 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T19:10:26.961 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T19:10:26.962 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T19:10:26.963 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T19:10:26.964 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T19:10:26.965 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T19:10:26.966 INFO:teuthology.orchestra.run.smithi053.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.967 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T19:10:26.968 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T19:10:26.969 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T19:10:26.970 INFO:teuthology.orchestra.run.smithi053.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T19:10:26.970 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T19:10:26.970 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T19:10:26.970 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T19:10:26.970 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:26.970 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-04-25T19:10:27.398 DEBUG:teuthology.parallel:result is None 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T19:10:27.713 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T19:10:27.714 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T19:10:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T19:10:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T19:10:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T19:10:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T19:10:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T19:10:27.715 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T19:10:27.726 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T19:10:27.727 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T19:10:27.728 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T19:10:27.729 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T19:10:27.730 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T19:10:27.731 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T19:10:27.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T19:10:27.733 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T19:10:27.734 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T19:10:27.735 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T19:10:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T19:10:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T19:10:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T19:10:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T19:10:27.736 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T19:10:29.997 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T19:10:29.998 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:29.999 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T19:10:30.000 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T19:10:30.001 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T19:10:30.002 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T19:10:30.003 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T19:10:30.004 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T19:10:30.005 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T19:10:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T19:10:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T19:10:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T19:10:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T19:10:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T19:10:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T19:10:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T19:10:30.006 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T19:10:30.007 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T19:10:30.008 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.009 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T19:10:30.009 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T19:10:30.009 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T19:10:30.009 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T19:10:30.009 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T19:10:30.009 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T19:10:30.010 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T19:10:30.011 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T19:10:30.011 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T19:10:30.011 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T19:10:30.011 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T19:10:30.011 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T19:10:30.012 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T19:10:30.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:30.012 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-25T19:10:30.359 DEBUG:teuthology.parallel:result is None 2024-04-25T19:10:30.360 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:10:30.507 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T19:10:30.529 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T19:10:30.529 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T19:10:30.529 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T19:10:30.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T19:10:30.674 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T19:10:30.701 INFO:teuthology.orchestra.run.smithi053.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T19:10:30.701 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T19:10:30.701 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T19:10:30.703 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T19:10:30.704 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:30.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T19:10:30.730 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:10:30.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T19:10:30.772 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T19:10:30.772 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:30.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T19:10:30.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T19:10:30.862 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:10:30.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T19:10:30.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T19:10:30.958 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T19:10:30.959 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:30.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T19:10:30.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T19:10:31.049 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:10:31.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T19:10:31.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T19:10:31.151 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T19:10:31.152 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:31.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T19:10:31.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T19:10:31.243 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:10:31.243 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T19:10:31.268 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T19:10:31.335 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T19:10:31.423 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T19:10:31.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-04-25T19:10:31.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2024-04-25T19:10:31.451 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T19:10:31.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T19:10:31.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T19:10:31.517 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T19:10:31.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T19:10:31.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T19:10:31.588 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T19:10:31.589 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T19:10:31.589 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T19:10:31.589 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:31.589 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T19:10:31.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T19:10:31.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T19:10:31.771 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:10:31.771 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T19:10:31.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T19:10:31.900 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T19:10:31.930 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T19:10:31.930 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T19:10:31.930 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302', 'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7673302', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240425.015853-debug'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1880, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'sleep_before_teardown': 0, 'subset': '178/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240425.015853-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJJmvhjPUNA6O+Zm951C9UJ8iIOyB9VdjG3iONnVc4gi+6f6U2s0ccHrA7ficl6kRUwDV54Q4Lv4VVGUye0/hiI=', 'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+uYEmH8wMWDuHceKaHtf+8QcIMfKsL70h7iFRIWpcEu0VUuc4Jxstes/4KeBy7tp+7jcuds47ka/VTX79sfao='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'MDS_FAILED', 'MDS_DAMAGE'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_recovery_pool']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_10:00:23', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T19:10:31.930 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T19:10:31.932 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T19:10:31.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T19:10:31.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T19:10:32.044 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:32.044 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T19:10:32.067 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T19:10:32.067 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-04-25T19:10:32.121 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T19:10:32.121 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:10:32.121 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-25T19:10:32.121 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:10:32.121 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:10:32.122 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 19:10:22.421447829 +0000 2024-04-25T19:10:32.122 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 19:08:30.778154966 +0000 2024-04-25T19:10:32.122 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 19:08:30.778154966 +0000 2024-04-25T19:10:32.122 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 19:08:30.778154966 +0000 2024-04-25T19:10:32.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T19:10:32.185 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-25T19:10:32.185 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-25T19:10:32.186 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 8.9497e-05 s, 5.7 MB/s 2024-04-25T19:10:32.187 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T19:10:32.243 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-04-25T19:10:32.298 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T19:10:32.299 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:10:32.299 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T19:10:32.299 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:10:32.299 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:10:32.299 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 19:10:22.422447801 +0000 2024-04-25T19:10:32.299 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 19:08:31.138144178 +0000 2024-04-25T19:10:32.299 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 19:08:31.138144178 +0000 2024-04-25T19:10:32.299 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 19:08:31.138144178 +0000 2024-04-25T19:10:32.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T19:10:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-25T19:10:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-25T19:10:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 8.9095e-05 s, 5.7 MB/s 2024-04-25T19:10:32.362 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T19:10:32.419 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 19:10:22.422447801 +0000 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 19:08:31.497133420 +0000 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 19:08:31.497133420 +0000 2024-04-25T19:10:32.477 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 19:08:31.497133420 +0000 2024-04-25T19:10:32.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T19:10:32.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-25T19:10:32.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-25T19:10:32.541 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 8.8737e-05 s, 5.8 MB/s 2024-04-25T19:10:32.543 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T19:10:32.598 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-04-25T19:10:32.653 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T19:10:32.653 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:10:32.653 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-25T19:10:32.653 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:10:32.654 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:10:32.654 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-25 19:10:22.423447772 +0000 2024-04-25T19:10:32.654 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-25 19:08:31.836123261 +0000 2024-04-25T19:10:32.654 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-25 19:08:31.836123261 +0000 2024-04-25T19:10:32.654 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-25 19:08:31.836123261 +0000 2024-04-25T19:10:32.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T19:10:32.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-25T19:10:32.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-25T19:10:32.716 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000102933 s, 5.0 MB/s 2024-04-25T19:10:32.717 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T19:10:32.774 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:32.775 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:10:32.775 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T19:10:32.794 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T19:10:32.794 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-25 19:10:19.345258314 +0000 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-25 19:08:30.852725747 +0000 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-25 19:08:30.852725747 +0000 2024-04-25T19:10:32.852 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-25 19:08:30.852725747 +0000 2024-04-25T19:10:32.852 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T19:10:32.920 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-25T19:10:32.920 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-25T19:10:32.920 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000119474 s, 4.3 MB/s 2024-04-25T19:10:32.921 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T19:10:32.980 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-04-25T19:10:33.038 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T19:10:33.038 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:10:33.038 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-25T19:10:33.038 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:10:33.038 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:10:33.038 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-25 19:10:19.346258295 +0000 2024-04-25T19:10:33.038 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-25 19:08:31.191718523 +0000 2024-04-25T19:10:33.038 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-25 19:08:31.191718523 +0000 2024-04-25T19:10:33.039 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-25 19:08:31.191718523 +0000 2024-04-25T19:10:33.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T19:10:33.104 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-25T19:10:33.104 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-25T19:10:33.104 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000111376 s, 4.6 MB/s 2024-04-25T19:10:33.105 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T19:10:33.163 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-04-25T19:10:33.220 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T19:10:33.220 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:10:33.220 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T19:10:33.221 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:10:33.221 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:10:33.221 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-25 19:10:19.346258295 +0000 2024-04-25T19:10:33.221 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-25 19:08:31.519711534 +0000 2024-04-25T19:10:33.221 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-25 19:08:31.519711534 +0000 2024-04-25T19:10:33.221 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-25 19:08:31.519711534 +0000 2024-04-25T19:10:33.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T19:10:33.286 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-25T19:10:33.286 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-25T19:10:33.286 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000106318 s, 4.8 MB/s 2024-04-25T19:10:33.287 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T19:10:33.345 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-04-25T19:10:33.402 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T19:10:33.402 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T19:10:33.402 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T19:10:33.402 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T19:10:33.402 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T19:10:33.403 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-04-25 19:10:19.346258295 +0000 2024-04-25T19:10:33.403 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-04-25 19:08:31.853704416 +0000 2024-04-25T19:10:33.403 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-04-25 19:08:31.853704416 +0000 2024-04-25T19:10:33.403 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-04-25 19:08:31.853704416 +0000 2024-04-25T19:10:33.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T19:10:33.467 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-04-25T19:10:33.468 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-04-25T19:10:33.468 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000109107 s, 4.7 MB/s 2024-04-25T19:10:33.469 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T19:10:33.526 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:33.527 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T19:10:33.527 INFO:tasks.ceph:Generating config... 2024-04-25T19:10:33.528 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T19:10:33.528 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T19:10:33.528 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T19:10:33.528 INFO:tasks.ceph:[client] pid file = 2024-04-25T19:10:33.528 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T19:10:33.528 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T19:10:33.528 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-25T19:10:33.528 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T19:10:33.528 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T19:10:33.529 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T19:10:33.530 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T19:10:33.530 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T19:10:33.530 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T19:10:33.530 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T19:10:33.530 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T19:10:33.530 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T19:10:33.530 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T19:10:33.530 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-25T19:10:33.530 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-25T19:10:33.530 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T19:10:33.531 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T19:10:33.531 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-25T19:10:33.531 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T19:10:33.531 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T19:10:33.531 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T19:10:33.531 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T19:10:33.531 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T19:10:33.531 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T19:10:33.531 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T19:10:33.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T19:10:33.592 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T19:10:33.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T19:10:33.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T19:10:33.670 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28'), ('mon.b', '172.21.15.53'), ('mon.c', '[v2:172.21.15.53:3301,v1:172.21.15.53:6790]')] 2024-04-25T19:10:33.671 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T19:10:33.672 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:33.672 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T19:10:33.725 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --add b 172.21.15.53 --addv c '[v2:172.21.15.53:3301,v1:172.21.15.53:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:33.822 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:33.822 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2024-04-25T19:10:33.821885+0000 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-25T19:10:33.821885+0000 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 17 (quincy) 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:1: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.b 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:2: [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] mon.c 2024-04-25T19:10:33.823 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T19:10:33.824 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T19:10:33.839 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5d8007d4-24a5-41be-aa4d-cd64fedac62a... 2024-04-25T19:10:33.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T19:10:33.882 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.922 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-04-25T19:10:33.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.924 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]" 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-04-25T19:10:33.925 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug trim objects = true 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.926 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 20 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 20 2024-04-25T19:10:33.927 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 10 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-04-25T19:10:33.928 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-04-25T19:10:33.929 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-04-25T19:10:33.930 INFO:teuthology.orchestra.run.smithi028.stdout: mon down mkfs grace = 300 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-04-25T19:10:33.931 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = True 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-04-25T19:10:33.932 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = True 2024-04-25T19:10:33.933 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-04-25T19:10:33.933 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-04-25T19:10:33.933 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-04-25T19:10:33.933 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-04-25T19:10:33.933 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-04-25T19:10:33.933 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T19:10:33.933 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-04-25T19:10:33.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T19:10:33.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "172.21.15.28,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]" 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on skipped message = False 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:10:33.936 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug trim objects = true 2024-04-25T19:10:33.937 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-04-25T19:10:33.938 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 20 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 20 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 10 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-04-25T19:10:33.940 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.941 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: mon down mkfs grace = 300 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: mon op complaint time = 120 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: client mount timeout = 600 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-04-25T19:10:33.942 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = "" 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a] 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c] 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b] 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d] 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout:[mds] 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds = 20 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds balancer = 20 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:10:33.943 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug frag = True 2024-04-25T19:10:33.944 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug scatterstat = True 2024-04-25T19:10:33.944 INFO:teuthology.orchestra.run.smithi053.stdout: mds op complaint time = 180 2024-04-25T19:10:33.944 INFO:teuthology.orchestra.run.smithi053.stdout: mds verify scatter = True 2024-04-25T19:10:33.944 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-04-25T19:10:33.944 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-04-25T19:10:33.944 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-04-25T19:10:33.945 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T19:10:33.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T19:10:34.076 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T19:10:34.076 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:34.076 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T19:10:34.091 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:34.091 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T19:10:34.146 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:10:34.147 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:34.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T19:10:34.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T19:10:34.220 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:10:34.220 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:34.275 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:10:34.276 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:10:34.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T19:10:34.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T19:10:34.317 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:10:34.317 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:34.373 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T19:10:34.374 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T19:10:34.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T19:10:34.433 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T19:10:34.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T19:10:34.505 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T19:10:34.509 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T19:10:34.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T19:10:34.569 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T19:10:34.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T19:10:34.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T19:10:34.694 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T19:10:34.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T19:10:34.721 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T19:10:34.785 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T19:10:34.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T19:10:34.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T19:10:34.914 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T19:10:34.916 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T19:10:34.941 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T19:10:34.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T19:10:34.991 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T19:10:35.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-25T19:10:35.053 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-25T19:10:35.067 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T19:10:35.116 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T19:10:35.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-25T19:10:35.178 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-25T19:10:35.191 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T19:10:35.191 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T19:10:35.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T19:10:35.215 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:35.215 INFO:tasks.ceph:role: osd.0 2024-04-25T19:10:35.215 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:10:35.215 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T19:10:35.279 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:10:35.279 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:10:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:10:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:10:35.280 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:10:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:10:35.280 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:10:35.280 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:10:35.280 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:10:35.280 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:10:35.315 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-04-25T19:10:35.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-04-25T19:10:35.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T19:10:35.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T19:10:35.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T19:10:35.520 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:35.520 INFO:tasks.ceph:role: osd.1 2024-04-25T19:10:35.520 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:10:35.521 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T19:10:35.584 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:10:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:10:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:10:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:10:35.584 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:10:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:10:35.585 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:10:35.585 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:10:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:10:35.585 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:10:35.616 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-04-25T19:10:35.618 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-04-25T19:10:35.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T19:10:35.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T19:10:35.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T19:10:35.819 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:35.820 INFO:tasks.ceph:role: osd.2 2024-04-25T19:10:35.820 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:10:35.820 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T19:10:35.883 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:10:35.883 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:10:35.883 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:10:35.883 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:10:35.883 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:10:35.883 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:10:35.883 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:10:35.884 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:10:35.884 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:10:35.884 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:10:35.916 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-04-25T19:10:35.918 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-04-25T19:10:35.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T19:10:35.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T19:10:36.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T19:10:36.117 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:36.117 INFO:tasks.ceph:role: osd.3 2024-04-25T19:10:36.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:10:36.117 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:10:36.182 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:10:36.183 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:10:36.214 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-04-25T19:10:36.216 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-04-25T19:10:36.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T19:10:36.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T19:10:36.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:36.459+0000 7f65f42b3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T19:10:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:36.460+0000 7f65f42b3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T19:10:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:36.460+0000 7f65f42b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T19:10:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:36.460+0000 7f65f42b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T19:10:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:36.460+0000 7f65f42b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T19:10:36.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:36.460+0000 7f65f42b3640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T19:10:39.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T19:10:39.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:39.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:39.453+0000 7fec4949b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T19:10:39.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:39.453+0000 7fec4949b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T19:10:39.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:39.453+0000 7fec4949b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T19:10:39.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:39.453+0000 7fec4949b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T19:10:39.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:39.453+0000 7fec4949b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T19:10:39.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:39.453+0000 7fec4949b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T19:10:42.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T19:10:42.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:42.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:42.433+0000 7f388a63d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T19:10:42.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:42.434+0000 7f388a63d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T19:10:42.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:42.434+0000 7f388a63d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T19:10:42.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:42.434+0000 7f388a63d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T19:10:42.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:42.434+0000 7f388a63d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T19:10:42.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:42.434+0000 7f388a63d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T19:10:45.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T19:10:45.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:45.413+0000 7f2d1a433640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T19:10:45.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:45.413+0000 7f2d1a433640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T19:10:45.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:45.413+0000 7f2d1a433640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T19:10:45.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:45.413+0000 7f2d1a433640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T19:10:45.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:45.413+0000 7f2d1a433640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T19:10:45.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:10:45.414+0000 7f2d1a433640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T19:10:48.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T19:10:48.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T19:10:48.323 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:48.323 INFO:tasks.ceph:role: osd.4 2024-04-25T19:10:48.323 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:10:48.323 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T19:10:48.393 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:10:48.394 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:10:48.425 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-04-25T19:10:48.428 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-04-25T19:10:48.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T19:10:48.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T19:10:48.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T19:10:48.632 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:48.632 INFO:tasks.ceph:role: osd.5 2024-04-25T19:10:48.632 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:10:48.633 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T19:10:48.697 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:10:48.698 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:10:48.698 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:10:48.698 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:10:48.698 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:10:48.698 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:10:48.698 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:10:48.698 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:10:48.698 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:10:48.699 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:10:48.730 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-04-25T19:10:48.732 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-04-25T19:10:48.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T19:10:48.806 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T19:10:48.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T19:10:48.940 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:48.940 INFO:tasks.ceph:role: osd.6 2024-04-25T19:10:48.940 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:10:48.941 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T19:10:49.004 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:10:49.004 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:10:49.004 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:10:49.004 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:10:49.004 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:10:49.004 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:10:49.004 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:10:49.005 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:10:49.005 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:10:49.005 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:10:49.036 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-04-25T19:10:49.038 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-04-25T19:10:49.038 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T19:10:49.106 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T19:10:49.174 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T19:10:49.238 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T19:10:49.238 INFO:tasks.ceph:role: osd.7 2024-04-25T19:10:49.238 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:10:49.238 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T19:10:49.301 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T19:10:49.302 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T19:10:49.335 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-04-25T19:10:49.337 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-04-25T19:10:49.337 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T19:10:49.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T19:10:49.472 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:49.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:49.578+0000 7f1d574e3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T19:10:49.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:49.578+0000 7f1d574e3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T19:10:49.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:49.578+0000 7f1d574e3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T19:10:49.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:49.578+0000 7f1d574e3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T19:10:49.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:49.578+0000 7f1d574e3640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T19:10:49.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:49.579+0000 7f1d574e3640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T19:10:52.433 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T19:10:52.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:52.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:52.564+0000 7fdf3752c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T19:10:52.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:52.564+0000 7fdf3752c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T19:10:52.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:52.564+0000 7fdf3752c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T19:10:52.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:52.564+0000 7fdf3752c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T19:10:52.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:52.564+0000 7fdf3752c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T19:10:52.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:52.565+0000 7fdf3752c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T19:10:55.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T19:10:55.457 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:55.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:55.563+0000 7ff788c32640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T19:10:55.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:55.563+0000 7ff788c32640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T19:10:55.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:55.563+0000 7ff788c32640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T19:10:55.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:55.563+0000 7ff788c32640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T19:10:55.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:55.563+0000 7ff788c32640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T19:10:55.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:55.563+0000 7ff788c32640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T19:10:58.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T19:10:58.443 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:10:58.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:58.543+0000 7f12f9424640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T19:10:58.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:58.543+0000 7f12f9424640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T19:10:58.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:58.543+0000 7f12f9424640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T19:10:58.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:58.543+0000 7f12f9424640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T19:10:58.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:58.543+0000 7f12f9424640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T19:10:58.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:10:58.543+0000 7f12f9424640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T19:11:01.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T19:11:01.423 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T19:11:01.423 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:01.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T19:11:01.453 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:01.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T19:11:01.523 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:01.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T19:11:01.592 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:01.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T19:11:01.660 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:01.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T19:11:01.729 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:01.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T19:11:01.795 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:01.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T19:11:01.861 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:01.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T19:11:01.890 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:01.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T19:11:01.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:11:01.934 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:11:01.960 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:01.961 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T19:11:02.028 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:02.028 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T19:11:02.095 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:02.095 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T19:11:02.161 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:02.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T19:11:02.226 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:02.227 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T19:11:02.291 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:02.291 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T19:11:02.308 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:02.308 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-25T19:11:02.364 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:02.364 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T19:11:02.380 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:02.380 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-25T19:11:02.436 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T19:11:02.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T19:11:02.438 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T19:11:02.478 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.y] 2024-04-25T19:11:02.478 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCqqipm7QHJGRAAsIrA+E/sN4HzAyzh3RIabg== 2024-04-25T19:11:02.478 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCqqipmiMziIRAAIlkDemOas1/wLLSfrPOtSw== 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCqqipmDDFcKRAA/e20s2LVvRm6r8j/elUESg== 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCsqipmfYl4GxAAhlhjSJHAr07miHRZvkrW/w== 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCvqipmgLkkGxAAKnKUs3oTOfZDKGe+Bd7MZg== 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCyqipm6RbtGRAA/vItjrpYyYv98tVL1xDwHw== 2024-04-25T19:11:02.479 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQC1qipmfYO1GBAA33Hurpq9u03VKZk2Hw99LA== 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCqqipmt2IdHhAAU7Iv0W1CnFpsi+zs3Zg2Yg== 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCqqipmSLzKLhAAlKSU261QzMzYnLfxJDAh6w== 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCqqipmOCFsNhAAFpeHA6Rfn9noLEk/t2iS7Q== 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.4] 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQC5qipmN5mHIhAAxNKhJie7i/i1ENVDbLN2/g== 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.5] 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQC8qipmOgu1IRAAnUsdR58QJkdmI71ugbA/8Q== 2024-04-25T19:11:02.480 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.6] 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQC/qipmbnihIRAA5M1YN90/yOeRMaDt6QW3Fg== 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.7] 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDCqipm1v9sIBAAEUE5Th5yvG3g2uVmioB2rw== 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCqqipmM9YPOxAAJwNJw9dfCED4wy79hvmLNg== 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout:[client.2] 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCrqipmv7UoAxAAMsoPXb/JY3YHQPtO+QE6Ew== 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout:[client.1] 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCrqipm5bLpBhAASSZf55x8RA98ZzfDf8pocw== 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout:[client.3] 2024-04-25T19:11:02.481 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCrqipmEWGTChAAso9pw1SJnc/t9bczpddQJg== 2024-04-25T19:11:02.498 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.y] 2024-04-25T19:11:02.498 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCqqipm7QHJGRAAsIrA+E/sN4HzAyzh3RIabg== 2024-04-25T19:11:02.498 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a] 2024-04-25T19:11:02.498 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCqqipmiMziIRAAIlkDemOas1/wLLSfrPOtSw== 2024-04-25T19:11:02.498 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c] 2024-04-25T19:11:02.498 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCqqipmDDFcKRAA/e20s2LVvRm6r8j/elUESg== 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCsqipmfYl4GxAAhlhjSJHAr07miHRZvkrW/w== 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCvqipmgLkkGxAAKnKUs3oTOfZDKGe+Bd7MZg== 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCyqipm6RbtGRAA/vItjrpYyYv98tVL1xDwHw== 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.3] 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC1qipmfYO1GBAA33Hurpq9u03VKZk2Hw99LA== 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCqqipmt2IdHhAAU7Iv0W1CnFpsi+zs3Zg2Yg== 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b] 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCqqipmSLzKLhAAlKSU261QzMzYnLfxJDAh6w== 2024-04-25T19:11:02.499 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d] 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCqqipmOCFsNhAAFpeHA6Rfn9noLEk/t2iS7Q== 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.4] 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC5qipmN5mHIhAAxNKhJie7i/i1ENVDbLN2/g== 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.5] 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC8qipmOgu1IRAAnUsdR58QJkdmI71ugbA/8Q== 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.6] 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC/qipmbnihIRAA5M1YN90/yOeRMaDt6QW3Fg== 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.7] 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDCqipm1v9sIBAAEUE5Th5yvG3g2uVmioB2rw== 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCqqipmM9YPOxAAJwNJw9dfCED4wy79hvmLNg== 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout:[client.2] 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCrqipmv7UoAxAAMsoPXb/JY3YHQPtO+QE6Ew== 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout:[client.1] 2024-04-25T19:11:02.500 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCrqipm5bLpBhAASSZf55x8RA98ZzfDf8pocw== 2024-04-25T19:11:02.501 INFO:teuthology.orchestra.run.smithi053.stdout:[client.3] 2024-04-25T19:11:02.501 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCrqipmEWGTChAAso9pw1SJnc/t9bczpddQJg== 2024-04-25T19:11:02.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:11:02.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:11:02.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:11:02.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:11:02.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:11:02.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:11:02.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:02.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:02.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:02.799 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:02.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:02.861 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:02.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:02.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:02.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:11:02.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T19:11:03.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:11:03.048 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:11:03.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:11:03.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T19:11:03.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:03.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:03.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:03.240 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:03.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:03.302 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:03.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:03.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T19:11:03.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:11:03.427 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:11:03.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:11:03.492 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:11:03.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:11:03.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:11:03.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:11:03.621 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T19:11:03.685 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T19:11:03.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T19:11:03.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T19:11:03.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T19:11:03.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T19:11:03.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T19:11:03.972 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T19:11:03.996 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T19:11:04.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T19:11:04.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T19:11:04.191 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:11:04.193 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T19:11:04.245 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T19:11:04.245 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T19:11:04.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T19:11:04.250 INFO:tasks.ceph.mon.a:Started 2024-04-25T19:11:04.250 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T19:11:04.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T19:11:04.288 INFO:tasks.ceph.mon.b:Started 2024-04-25T19:11:04.288 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T19:11:04.288 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T19:11:04.290 INFO:tasks.ceph.mon.c:Started 2024-04-25T19:11:04.290 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T19:11:04.290 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T19:11:04.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T19:11:04.293 INFO:tasks.ceph.mgr.y:Started 2024-04-25T19:11:04.293 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T19:11:04.293 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T19:11:04.295 INFO:tasks.ceph.mgr.x:Started 2024-04-25T19:11:04.295 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T19:11:04.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T19:11:04.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.407+0000 7f508a934640 1 Processor -- start 2024-04-25T19:11:04.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.407+0000 7f508a934640 1 -- start start 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.408+0000 7f508a934640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 0x7f508406dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.408+0000 7f508a934640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f50840b4110 0x7f50840b4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.408+0000 7f508a934640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f508406f000 con 0x7f508406e4e0 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.408+0000 7f508a934640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f508406ea40 con 0x7f508415aa00 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.408+0000 7f508a934640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f508406ed20 con 0x7f50840b4110 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.408+0000 7f5083fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 0x7f508406dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.408+0000 7f5083fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 0x7f508406dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37884/0 (socket says 172.21.15.28:37884) 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.408+0000 7f5083fff640 1 -- 172.21.15.28:0/2366217273 learned_addr learned my addr 172.21.15.28:0/2366217273 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5083fff640 1 -- 172.21.15.28:0/2366217273 >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 msgr2=0x7f508406dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5083fff640 1 -- 172.21.15.28:0/2366217273 >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 msgr2=0x7f508406dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T19:11:04.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5083fff640 1 --2- 172.21.15.28:0/2366217273 >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 0x7f508406dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:11:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5083fff640 1 --2- 172.21.15.28:0/2366217273 >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 0x7f508406dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:11:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5088ed1640 1 --2- 172.21.15.28:0/2366217273 >> v2:172.21.15.53:3300/0 conn(0x7f50840b4110 0x7f50840b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5088ed1640 1 -- 172.21.15.28:0/2366217273 >> v2:172.21.15.53:3300/0 conn(0x7f50840b4110 msgr2=0x7f50840b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T19:11:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5088ed1640 1 -- 172.21.15.28:0/2366217273 >> v2:172.21.15.53:3300/0 conn(0x7f50840b4110 msgr2=0x7f50840b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T19:11:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5088ed1640 1 --2- 172.21.15.28:0/2366217273 >> v2:172.21.15.53:3300/0 conn(0x7f50840b4110 0x7f50840b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T19:11:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.409+0000 7f5088ed1640 1 --2- 172.21.15.28:0/2366217273 >> v2:172.21.15.53:3300/0 conn(0x7f50840b4110 0x7f50840b4520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.567+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 746653938 0 0) 0x7f508406f000 con 0x7f508406e4e0 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.567+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f506c003520 con 0x7f508406e4e0 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 <== mon.1 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2389990972 0 0) 0x7f506c003520 con 0x7f508406e4e0 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 >> v2:172.21.15.53:3300/0 conn(0x7f50840b4110 msgr2=0x7f50840b4520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 --2- 172.21.15.28:0/2366217273 >> v2:172.21.15.53:3300/0 conn(0x7f50840b4110 0x7f50840b4520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 msgr2=0x7f508406dfa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 --2- 172.21.15.28:0/2366217273 >> v2:172.21.15.28:3300/0 conn(0x7f508415aa00 0x7f508406dfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50840b27e0 con 0x7f508406e4e0 2024-04-25T19:11:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 <== mon.1 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f5078001670 con 0x7f508406e4e0 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 >> v1:172.21.15.53:6789/0 conn(0x7f508406e4e0 legacy=0x7f50840b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f506c0039b0 0x7f506c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f506c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f506c005220 0x7f506c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f508406ea40 con 0x7f506c004800 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f508406ed20 con 0x7f506c005220 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f506c00c230 con 0x7f506c0039b0 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f50837fe640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f506c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5083fff640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f506c0039b0 0x7f506c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5088ed1640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f506c005220 0x7f506c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f50837fe640 1 -- 172.21.15.28:0/2366217273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f506c0039b0 msgr2=0x7f506c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f50837fe640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f506c0039b0 0x7f506c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f50837fe640 1 -- 172.21.15.28:0/2366217273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f506c005220 msgr2=0x7f506c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f50837fe640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f506c005220 0x7f506c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f5088ed1640 1 -- 172.21.15.28:0/2366217273 reap_dead start 2024-04-25T19:11:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.568+0000 7f50837fe640 1 -- 172.21.15.28:0/2366217273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f506c003520 con 0x7f506c004800 2024-04-25T19:11:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.569+0000 7f50837fe640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f506c004c10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f507800c190 tx=0x7f506c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be20aee7aef61747 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.571+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078036070 con 0x7f506c004800 2024-04-25T19:11:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.571+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5078031600 con 0x7f506c004800 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.571+0000 7f5082ffd640 1 -- 172.21.15.28:0/2366217273 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078007c60 con 0x7f506c004800 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.572+0000 7f508a934640 1 -- 172.21.15.28:0/2366217273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 msgr2=0x7f506c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.572+0000 7f508a934640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f506c004c10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f507800c190 tx=0x7f506c005830 comp rx=0 tx=0).stop 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.572+0000 7f508a934640 1 -- 172.21.15.28:0/2366217273 shutdown_connections 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.572+0000 7f508a934640 1 --2- 172.21.15.28:0/2366217273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f506c004c10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.572+0000 7f508a934640 1 -- 172.21.15.28:0/2366217273 >> 172.21.15.28:0/2366217273 conn(0x7f50840b75d0 msgr2=0x7f50840b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.572+0000 7f508a934640 1 -- 172.21.15.28:0/2366217273 shutdown_connections 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.572+0000 7f508a934640 1 -- 172.21.15.28:0/2366217273 wait complete. 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.574+0000 7f508a934640 1 Processor -- start 2024-04-25T19:11:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.574+0000 7f508a934640 1 -- start start 2024-04-25T19:11:04.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f508a934640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f5084157a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:04.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f508a934640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5084152db0 0x7f5084152720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f508a934640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50841f1ab0 0x7f50841f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f508a934640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f508415ae10 con 0x7f506c004800 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f508a934640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f508415d7d0 con 0x7f5084152db0 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f508a934640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f508406dfa0 con 0x7f50841f1ab0 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f5083fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f5084157a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f5083fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f5084157a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37900/0 (socket says 172.21.15.28:37900) 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f5083fff640 1 -- 172.21.15.28:0/3611642155 learned_addr learned my addr 172.21.15.28:0/3611642155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.575+0000 7f50837fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5084152db0 0x7f5084152720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5088ed1640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50841f1ab0 0x7f50841f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5083fff640 1 -- 172.21.15.28:0/3611642155 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50841f1ab0 msgr2=0x7f50841f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5083fff640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50841f1ab0 0x7f50841f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5083fff640 1 -- 172.21.15.28:0/3611642155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5084152db0 msgr2=0x7f5084152720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:04.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5083fff640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5084152db0 0x7f5084152720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5083fff640 1 -- 172.21.15.28:0/3611642155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50840cfb50 con 0x7f506c004800 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f50837fe640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5084152db0 0x7f5084152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5083fff640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f5084157a30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f50740119e0 tx=0x7f5074011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5080ff9640 1 -- 172.21.15.28:0/3611642155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5074010ba0 con 0x7f506c004800 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5080ff9640 1 -- 172.21.15.28:0/3611642155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5074010d40 con 0x7f506c004800 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5080ff9640 1 -- 172.21.15.28:0/3611642155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507401b640 con 0x7f506c004800 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50840cfe30 con 0x7f506c004800 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50841535a0 con 0x7f506c004800 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5080ff9640 1 -- 172.21.15.28:0/3611642155 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f507401b7e0 con 0x7f506c004800 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.576+0000 7f5080ff9640 1 -- 172.21.15.28:0/3611642155 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f507400e400 con 0x7f506c004800 2024-04-25T19:11:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.577+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508406d830 con 0x7f506c004800 2024-04-25T19:11:04.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.580+0000 7f5080ff9640 1 -- 172.21.15.28:0/3611642155 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5074024440 con 0x7f506c004800 2024-04-25T19:11:04.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.648+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f50840d0350 con 0x7f506c004800 2024-04-25T19:11:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.650+0000 7f5080ff9640 1 -- 172.21.15.28:0/3611642155 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f5074024620 con 0x7f506c004800 2024-04-25T19:11:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 msgr2=0x7f5084157a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f5084157a30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f50740119e0 tx=0x7f5074011ea0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 shutdown_connections 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50841f1ab0 0x7f50841f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5084152db0 0x7f5084152720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 --2- 172.21.15.28:0/3611642155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f506c004800 0x7f5084157a30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 >> 172.21.15.28:0/3611642155 conn(0x7f50840b75d0 msgr2=0x7f5084071330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 shutdown_connections 2024-04-25T19:11:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:04.651+0000 7f508a934640 1 -- 172.21.15.28:0/3611642155 wait complete. 2024-04-25T19:11:04.667 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T19:11:04.668 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T19:11:04.668 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:04.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T19:11:04.682 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:04.680+0000 7f294431c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:11:04.692 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:04.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T19:11:04.698 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:04.696+0000 7f69f4172040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T19:11:04.721 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:04.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T19:11:04.786 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:04.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T19:11:04.845 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:04.843+0000 7f294431c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:11:04.851 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:04.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T19:11:04.852 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:04.851+0000 7f69f4172040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:11:04.875 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:04.875 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T19:11:04.902 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:04.900+0000 7f294431c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:11:04.908 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:04.907+0000 7f69f4172040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T19:11:04.925 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:04.925 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T19:11:04.989 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:11:04.989 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T19:11:05.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd 0 2024-04-25T19:11:05.150 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:05.148+0000 7f69f4172040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:11:05.162 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:05.160+0000 7f294431c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T19:11:05.198 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:05.196+0000 7f69f4172040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:11:05.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.204+0000 7fed024f9640 1 Processor -- start 2024-04-25T19:11:05.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.204+0000 7fed024f9640 1 -- start start 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.205+0000 7fed024f9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fecfc0b30e0 0x7fecfc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.205+0000 7fed024f9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fecfc09b540 0x7fecfc09b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.205+0000 7fed024f9640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecfc09be90 con 0x7fecfc161cf0 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.205+0000 7fed024f9640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fecfc0b3650 con 0x7fecfc0b30e0 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.205+0000 7fed024f9640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fecfc06c710 con 0x7fecfc09b540 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.205+0000 7fecfb7fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fecfc09b540 0x7fecfc09b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.205+0000 7fed00a96640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fecfc161cf0 0x7fecfc162100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49088/0 (socket says 172.21.15.53:49088) 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.205+0000 7fed00a96640 1 -- 172.21.15.53:0/2207980202 learned_addr learned my addr 172.21.15.53:0/2207980202 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.206+0000 7fecfbfff640 1 --2- 172.21.15.53:0/2207980202 >> v2:172.21.15.28:3300/0 conn(0x7fecfc0b30e0 0x7fecfc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.206+0000 7fecfb7fe640 1 -- 172.21.15.53:0/2207980202 >> v2:172.21.15.28:3300/0 conn(0x7fecfc0b30e0 msgr2=0x7fecfc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.206+0000 7fecfb7fe640 1 --2- 172.21.15.53:0/2207980202 >> v2:172.21.15.28:3300/0 conn(0x7fecfc0b30e0 0x7fecfc0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.206+0000 7fecfb7fe640 1 -- 172.21.15.53:0/2207980202 >> v1:172.21.15.53:6789/0 conn(0x7fecfc161cf0 legacy=0x7fecfc162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.206+0000 7fecfb7fe640 1 -- 172.21.15.53:0/2207980202 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecfc06c9f0 con 0x7fecfc09b540 2024-04-25T19:11:05.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.206+0000 7fecfb7fe640 1 --2- 172.21.15.53:0/2207980202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc09b540 0x7fecfc09b950 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fecfc09c4e0 tx=0x7fecf002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99d125c4fccb2344 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:05.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.207+0000 7fecfaffd640 1 -- 172.21.15.53:0/2207980202 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0037070 con 0x7fecfc09b540 2024-04-25T19:11:05.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.207+0000 7fecfaffd640 1 -- 172.21.15.53:0/2207980202 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecf0032440 con 0x7fecfc09b540 2024-04-25T19:11:05.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.207+0000 7fecfaffd640 1 -- 172.21.15.53:0/2207980202 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf002ae60 con 0x7fecfc09b540 2024-04-25T19:11:05.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.207+0000 7fed024f9640 1 -- 172.21.15.53:0/2207980202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc09b540 msgr2=0x7fecfc09b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.207+0000 7fed024f9640 1 --2- 172.21.15.53:0/2207980202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc09b540 0x7fecfc09b950 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fecfc09c4e0 tx=0x7fecf002a110 comp rx=0 tx=0).stop 2024-04-25T19:11:05.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.208+0000 7fed024f9640 1 -- 172.21.15.53:0/2207980202 shutdown_connections 2024-04-25T19:11:05.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.208+0000 7fed024f9640 1 --2- 172.21.15.53:0/2207980202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc09b540 0x7fecfc09b950 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.208+0000 7fed024f9640 1 --2- 172.21.15.53:0/2207980202 >> v2:172.21.15.28:3300/0 conn(0x7fecfc0b30e0 0x7fecfc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.208+0000 7fed024f9640 1 -- 172.21.15.53:0/2207980202 >> 172.21.15.53:0/2207980202 conn(0x7fecfc0b7620 msgr2=0x7fecfc0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:05.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.208+0000 7fed024f9640 1 -- 172.21.15.53:0/2207980202 shutdown_connections 2024-04-25T19:11:05.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.208+0000 7fed024f9640 1 -- 172.21.15.53:0/2207980202 wait complete. 2024-04-25T19:11:05.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.209+0000 7fed024f9640 1 Processor -- start 2024-04-25T19:11:05.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.210+0000 7fed024f9640 1 -- start start 2024-04-25T19:11:05.212 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:05.210+0000 7f294431c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T19:11:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fed024f9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fecfc09b540 0x7fecfc15c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fed024f9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc0b30e0 0x7fecfc1f8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fed024f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc161cf0 0x7fecfc1f8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fed024f9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc0cfe90 con 0x7fecfc161cf0 2024-04-25T19:11:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fed024f9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fecfc0cfd10 con 0x7fecfc0b30e0 2024-04-25T19:11:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fed024f9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fecfc161940 con 0x7fecfc09b540 2024-04-25T19:11:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfbfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fecfc09b540 0x7fecfc15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfb7fe640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc0b30e0 0x7fecfc1f8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfbfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fecfc09b540 0x7fecfc15c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51568/0 (socket says 172.21.15.53:51568) 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfbfff640 1 -- 172.21.15.53:0/904635883 learned_addr learned my addr 172.21.15.53:0/904635883 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fed00a96640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc161cf0 0x7fecfc1f8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfb7fe640 1 -- 172.21.15.53:0/904635883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fecfc09b540 msgr2=0x7fecfc15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfb7fe640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fecfc09b540 0x7fecfc15c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfb7fe640 1 -- 172.21.15.53:0/904635883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc161cf0 msgr2=0x7fecfc1f8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfb7fe640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc161cf0 0x7fecfc1f8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfb7fe640 1 -- 172.21.15.53:0/904635883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecfc09e9b0 con 0x7fecfc0b30e0 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.211+0000 7fecfbfff640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fecfc09b540 0x7fecfc15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.212+0000 7fecfb7fe640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc0b30e0 0x7fecfc1f8020 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fecf0031000 tx=0x7fecf00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.212+0000 7fecf8ff9640 1 -- 172.21.15.53:0/904635883 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0041030 con 0x7fecfc0b30e0 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.212+0000 7fecf8ff9640 1 -- 172.21.15.53:0/904635883 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecf0030500 con 0x7fecfc0b30e0 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.212+0000 7fecf8ff9640 1 -- 172.21.15.53:0/904635883 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecf0037040 con 0x7fecfc0b30e0 2024-04-25T19:11:05.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.212+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecfc09ec90 con 0x7fecfc0b30e0 2024-04-25T19:11:05.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.212+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecfc0c8de0 con 0x7fecfc0b30e0 2024-04-25T19:11:05.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.212+0000 7fecf8ff9640 1 -- 172.21.15.53:0/904635883 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fecf0044070 con 0x7fecfc0b30e0 2024-04-25T19:11:05.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.212+0000 7fecf8ff9640 1 -- 172.21.15.53:0/904635883 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fecf003db50 con 0x7fecfc0b30e0 2024-04-25T19:11:05.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.213+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecfc050740 con 0x7fecfc0b30e0 2024-04-25T19:11:05.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.217+0000 7fecf8ff9640 1 -- 172.21.15.53:0/904635883 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fecf003dda0 con 0x7fecfc0b30e0 2024-04-25T19:11:05.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.291+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd", "id": 0} v 0) -- 0x7fecfc0c76a0 con 0x7fecfc0b30e0 2024-04-25T19:11:05.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.296+0000 7fecf8ff9640 1 -- 172.21.15.53:0/904635883 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fecf00308d0 con 0x7fecfc0b30e0 2024-04-25T19:11:05.297 INFO:teuthology.orchestra.run.smithi053.stdout:0 2024-04-25T19:11:05.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.296+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc0b30e0 msgr2=0x7fecfc1f8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.296+0000 7fed024f9640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc0b30e0 0x7fecfc1f8020 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fecf0031000 tx=0x7fecf00046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.296+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 shutdown_connections 2024-04-25T19:11:05.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.296+0000 7fed024f9640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc161cf0 0x7fecfc1f8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.296+0000 7fed024f9640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecfc0b30e0 0x7fecfc1f8020 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.296+0000 7fed024f9640 1 --2- 172.21.15.53:0/904635883 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fecfc09b540 0x7fecfc15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.297+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 >> 172.21.15.53:0/904635883 conn(0x7fecfc0b7620 msgr2=0x7fecfc071400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:05.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.297+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 shutdown_connections 2024-04-25T19:11:05.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.297+0000 7fed024f9640 1 -- 172.21.15.53:0/904635883 wait complete. 2024-04-25T19:11:05.309 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 19c91f41-b727-4e4a-a862-4a042b38ef05 1 2024-04-25T19:11:05.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.419+0000 7f273c31f640 1 Processor -- start 2024-04-25T19:11:05.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.419+0000 7f273c31f640 1 -- start start 2024-04-25T19:11:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f273c31f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27340cfa00 0x7f273415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f273c31f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f273409b540 0x7f273409b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f273c31f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f273409be90 con 0x7f27340b21d0 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f273c31f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f27340cffe0 con 0x7f27340cfa00 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f273c31f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f27340d02c0 con 0x7f273409b540 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f273a0bb640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27340cfa00 0x7f273415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f273a8bc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f273409b540 0x7f273409b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f27398ba640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f27340b21d0 0x7f2734157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49092/0 (socket says 172.21.15.53:49092) 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.420+0000 7f27398ba640 1 -- 172.21.15.53:0/1062819870 learned_addr learned my addr 172.21.15.53:0/1062819870 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.421+0000 7f273a0bb640 1 -- 172.21.15.53:0/1062819870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f273409b540 msgr2=0x7f273409b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.421+0000 7f273a0bb640 1 --2- 172.21.15.53:0/1062819870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f273409b540 0x7f273409b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.421+0000 7f273a0bb640 1 -- 172.21.15.53:0/1062819870 >> v1:172.21.15.53:6789/0 conn(0x7f27340b21d0 legacy=0x7f2734157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.421+0000 7f27390b9640 1 -- 172.21.15.53:0/1062819870 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559645997 0 0) 0x7f2724001110 con 0x7f27340b21d0 2024-04-25T19:11:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.421+0000 7f273a0bb640 1 -- 172.21.15.53:0/1062819870 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273415da20 con 0x7f27340cfa00 2024-04-25T19:11:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.421+0000 7f273a8bc640 1 --2- 172.21.15.53:0/1062819870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f273409b540 0x7f273409b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.422+0000 7f273a0bb640 1 --2- 172.21.15.53:0/1062819870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27340cfa00 0x7f273415e220 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f27300097a0 tx=0x7f273002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65cac75430a00ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.422+0000 7f27390b9640 1 -- 172.21.15.53:0/1062819870 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730037070 con 0x7f27340cfa00 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.422+0000 7f27390b9640 1 -- 172.21.15.53:0/1062819870 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f273002ae50 con 0x7f27340cfa00 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.422+0000 7f27390b9640 1 -- 172.21.15.53:0/1062819870 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730032660 con 0x7f27340cfa00 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.422+0000 7f273c31f640 1 -- 172.21.15.53:0/1062819870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27340cfa00 msgr2=0x7f273415e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.422+0000 7f273c31f640 1 --2- 172.21.15.53:0/1062819870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27340cfa00 0x7f273415e220 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f27300097a0 tx=0x7f273002a100 comp rx=0 tx=0).stop 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.423+0000 7f273c31f640 1 -- 172.21.15.53:0/1062819870 shutdown_connections 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.423+0000 7f273c31f640 1 --2- 172.21.15.53:0/1062819870 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f273409b540 0x7f273409b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.423+0000 7f273c31f640 1 --2- 172.21.15.53:0/1062819870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27340cfa00 0x7f273415e220 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.423+0000 7f273c31f640 1 -- 172.21.15.53:0/1062819870 >> 172.21.15.53:0/1062819870 conn(0x7f27340b7620 msgr2=0x7f27340b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.423+0000 7f273c31f640 1 -- 172.21.15.53:0/1062819870 shutdown_connections 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.423+0000 7f273c31f640 1 -- 172.21.15.53:0/1062819870 wait complete. 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.424+0000 7f273c31f640 1 Processor -- start 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.424+0000 7f273c31f640 1 -- start start 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273c31f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f273409b540 0x7f27340d1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273c31f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27340b21d0 0x7f27340d2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273c31f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27340cfa00 0x7f27341f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273c31f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27340b2760 con 0x7f27340cfa00 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273c31f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f27340b25e0 con 0x7f273409b540 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273c31f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f273406d680 con 0x7f27340b21d0 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f273409b540 0x7f27340d1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f273409b540 0x7f27340d1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38504/0 (socket says 172.21.15.53:38504) 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f27398ba640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27340b21d0 0x7f27340d2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 -- 172.21.15.53:0/3147950099 learned_addr learned my addr 172.21.15.53:0/3147950099 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a8bc640 1 --2- 172.21.15.53:0/3147950099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27340cfa00 0x7f27341f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 -- 172.21.15.53:0/3147950099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27340b21d0 msgr2=0x7f27340d2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 --2- 172.21.15.53:0/3147950099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27340b21d0 0x7f27340d2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 -- 172.21.15.53:0/3147950099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27340cfa00 msgr2=0x7f27341f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 --2- 172.21.15.53:0/3147950099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27340cfa00 0x7f27341f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 -- 172.21.15.53:0/3147950099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27340b0530 con 0x7f273409b540 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f273a0bb640 1 --2- 172.21.15.53:0/3147950099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f273409b540 0x7f27340d1190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2730035040 tx=0x7f27300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.426+0000 7f272affd640 1 -- 172.21.15.53:0/3147950099 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730041070 con 0x7f273409b540 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.427+0000 7f272affd640 1 -- 172.21.15.53:0/3147950099 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f273003c460 con 0x7f273409b540 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.427+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27340b0810 con 0x7f273409b540 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.427+0000 7f272affd640 1 -- 172.21.15.53:0/3147950099 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2730037040 con 0x7f273409b540 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.427+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27340b0dd0 con 0x7f273409b540 2024-04-25T19:11:05.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.427+0000 7f272affd640 1 -- 172.21.15.53:0/3147950099 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f273003a070 con 0x7f273409b540 2024-04-25T19:11:05.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.427+0000 7f272affd640 1 -- 172.21.15.53:0/3147950099 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f2730045ac0 con 0x7f273409b540 2024-04-25T19:11:05.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.428+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2734050740 con 0x7f273409b540 2024-04-25T19:11:05.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.430+0000 7f272affd640 1 -- 172.21.15.53:0/3147950099 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f273003c870 con 0x7f273409b540 2024-04-25T19:11:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.503+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "19c91f41-b727-4e4a-a862-4a042b38ef05", "id": 1} v 0) -- 0x7f273409bc70 con 0x7f273409b540 2024-04-25T19:11:05.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.507+0000 7f272affd640 1 -- 172.21.15.53:0/3147950099 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "19c91f41-b727-4e4a-a862-4a042b38ef05", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f273003ca50 con 0x7f273409b540 2024-04-25T19:11:05.509 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-04-25T19:11:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f273409b540 msgr2=0x7f27340d1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 --2- 172.21.15.53:0/3147950099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f273409b540 0x7f27340d1190 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2730035040 tx=0x7f27300046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 shutdown_connections 2024-04-25T19:11:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 --2- 172.21.15.53:0/3147950099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27340cfa00 0x7f27341f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 --2- 172.21.15.53:0/3147950099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27340b21d0 0x7f27340d2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 --2- 172.21.15.53:0/3147950099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f273409b540 0x7f27340d1190 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 >> 172.21.15.53:0/3147950099 conn(0x7f27340b7620 msgr2=0x7f273415e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 shutdown_connections 2024-04-25T19:11:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.508+0000 7f273c31f640 1 -- 172.21.15.53:0/3147950099 wait complete. 2024-04-25T19:11:05.520 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 6f313a73-d3d6-48af-bc8c-6d9faabc7b7f 2 2024-04-25T19:11:05.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.631+0000 7f319bf72640 1 Processor -- start 2024-04-25T19:11:05.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.631+0000 7f319bf72640 1 -- start start 2024-04-25T19:11:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.632+0000 7f319bf72640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31940b48a0 0x7f319409bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.632+0000 7f319bf72640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f31940b4e60 0x7f319406e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.632+0000 7f319bf72640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31940b3ca0 con 0x7f319409c2e0 2024-04-25T19:11:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.632+0000 7f319bf72640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f319409d390 con 0x7f31940b4e60 2024-04-25T19:11:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.632+0000 7f319bf72640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f31940b3f40 con 0x7f31940b48a0 2024-04-25T19:11:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.633+0000 7f3199d0e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31940b48a0 0x7f319409bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.633+0000 7f3199d0e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31940b48a0 0x7f319409bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51594/0 (socket says 172.21.15.53:51594) 2024-04-25T19:11:05.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.633+0000 7f3199d0e640 1 -- 172.21.15.53:0/1660575102 learned_addr learned my addr 172.21.15.53:0/1660575102 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:05.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.633+0000 7f319a50f640 1 --2- 172.21.15.53:0/1660575102 >> v2:172.21.15.28:3300/0 conn(0x7f31940b4e60 0x7f319406e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.633+0000 7f3199d0e640 1 -- 172.21.15.53:0/1660575102 >> v2:172.21.15.28:3300/0 conn(0x7f31940b4e60 msgr2=0x7f319406e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:05.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.633+0000 7f3199d0e640 1 --2- 172.21.15.53:0/1660575102 >> v2:172.21.15.28:3300/0 conn(0x7f31940b4e60 0x7f319406e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.633+0000 7f3199d0e640 1 -- 172.21.15.53:0/1660575102 >> v1:172.21.15.28:6789/0 conn(0x7f319409c2e0 legacy=0x7f319406da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.633+0000 7f3199d0e640 1 -- 172.21.15.53:0/1660575102 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31940b4220 con 0x7f31940b48a0 2024-04-25T19:11:05.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.634+0000 7f3199d0e640 1 --2- 172.21.15.53:0/1660575102 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31940b48a0 0x7f319409bda0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3190009870 tx=0x7f319002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe9239eee9ede359 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:05.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.634+0000 7f3198d0c640 1 -- 172.21.15.53:0/1660575102 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190037070 con 0x7f31940b48a0 2024-04-25T19:11:05.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.634+0000 7f3198d0c640 1 -- 172.21.15.53:0/1660575102 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3190032440 con 0x7f31940b48a0 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.634+0000 7f3198d0c640 1 -- 172.21.15.53:0/1660575102 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3190032810 con 0x7f31940b48a0 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.635+0000 7f319bf72640 1 -- 172.21.15.53:0/1660575102 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31940b48a0 msgr2=0x7f319409bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.635+0000 7f319bf72640 1 --2- 172.21.15.53:0/1660575102 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31940b48a0 0x7f319409bda0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3190009870 tx=0x7f319002a270 comp rx=0 tx=0).stop 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.635+0000 7f319bf72640 1 -- 172.21.15.53:0/1660575102 shutdown_connections 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.635+0000 7f319bf72640 1 --2- 172.21.15.53:0/1660575102 >> v2:172.21.15.28:3300/0 conn(0x7f31940b4e60 0x7f319406e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.635+0000 7f319bf72640 1 --2- 172.21.15.53:0/1660575102 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31940b48a0 0x7f319409bda0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.635+0000 7f319bf72640 1 -- 172.21.15.53:0/1660575102 >> 172.21.15.53:0/1660575102 conn(0x7f31940b5480 msgr2=0x7f31940b5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.635+0000 7f319bf72640 1 -- 172.21.15.53:0/1660575102 shutdown_connections 2024-04-25T19:11:05.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.635+0000 7f319bf72640 1 -- 172.21.15.53:0/1660575102 wait complete. 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.637+0000 7f319bf72640 1 Processor -- start 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.637+0000 7f319bf72640 1 -- start start 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319bf72640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f319409c2e0 0x7f31940cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319bf72640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31940b48a0 0x7f31941dac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319bf72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31940b4e60 0x7f31940b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319bf72640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f319409cfb0 con 0x7f31940b4e60 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319bf72640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f31940b3960 con 0x7f31940b48a0 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319bf72640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f319409d130 con 0x7f319409c2e0 2024-04-25T19:11:05.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f3199d0e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f319409c2e0 0x7f31940cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319950d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31940b48a0 0x7f31941dac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f3199d0e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f319409c2e0 0x7f31940cbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51596/0 (socket says 172.21.15.53:51596) 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f3199d0e640 1 -- 172.21.15.53:0/997612509 learned_addr learned my addr 172.21.15.53:0/997612509 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319a50f640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31940b4e60 0x7f31940b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319950d640 1 -- 172.21.15.53:0/997612509 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f319409c2e0 msgr2=0x7f31940cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319950d640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f319409c2e0 0x7f31940cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319950d640 1 -- 172.21.15.53:0/997612509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31940b4e60 msgr2=0x7f31940b7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319950d640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31940b4e60 0x7f31940b7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f319950d640 1 -- 172.21.15.53:0/997612509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31940cd830 con 0x7f31940b48a0 2024-04-25T19:11:05.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.638+0000 7f3199d0e640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f319409c2e0 0x7f31940cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.639+0000 7f319950d640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31940b48a0 0x7f31941dac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f318800c3c0 tx=0x7f318800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.639+0000 7f31867fc640 1 -- 172.21.15.53:0/997612509 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31880052d0 con 0x7f31940b48a0 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.639+0000 7f31867fc640 1 -- 172.21.15.53:0/997612509 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3188005490 con 0x7f31940b48a0 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.639+0000 7f31867fc640 1 -- 172.21.15.53:0/997612509 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3188007260 con 0x7f31940b48a0 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.639+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31940cc790 con 0x7f31940b48a0 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.639+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31940ccd50 con 0x7f31940b48a0 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.639+0000 7f31867fc640 1 -- 172.21.15.53:0/997612509 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3188007400 con 0x7f31940b48a0 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.639+0000 7f31867fc640 1 -- 172.21.15.53:0/997612509 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f31880035f0 con 0x7f31940b48a0 2024-04-25T19:11:05.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.640+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31940cc4b0 con 0x7f31940b48a0 2024-04-25T19:11:05.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.644+0000 7f31867fc640 1 -- 172.21.15.53:0/997612509 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f3188003a50 con 0x7f31940b48a0 2024-04-25T19:11:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.719+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6f313a73-d3d6-48af-bc8c-6d9faabc7b7f", "id": 2} v 0) -- 0x7f319406e720 con 0x7f31940b48a0 2024-04-25T19:11:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.722+0000 7f31867fc640 1 -- 172.21.15.53:0/997612509 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f313a73-d3d6-48af-bc8c-6d9faabc7b7f", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3188003d10 con 0x7f31940b48a0 2024-04-25T19:11:05.724 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-04-25T19:11:05.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31940b48a0 msgr2=0x7f31941dac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31940b48a0 0x7f31941dac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f318800c3c0 tx=0x7f318800c880 comp rx=0 tx=0).stop 2024-04-25T19:11:05.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 shutdown_connections 2024-04-25T19:11:05.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31940b4e60 0x7f31940b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31940b48a0 0x7f31941dac60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 --2- 172.21.15.53:0/997612509 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f319409c2e0 0x7f31940cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 >> 172.21.15.53:0/997612509 conn(0x7f31940b5480 msgr2=0x7f31940b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:05.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 shutdown_connections 2024-04-25T19:11:05.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.723+0000 7f319bf72640 1 -- 172.21.15.53:0/997612509 wait complete. 2024-04-25T19:11:05.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 420b3523-29ab-4043-bc1b-2457f61a5e04 3 2024-04-25T19:11:05.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.844+0000 7f80b70b1640 1 Processor -- start 2024-04-25T19:11:05.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.845+0000 7f80b70b1640 1 -- start start 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80b70b1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b00b2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80b70b1640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f80b01618a0 0x7f80b0161cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80b70b1640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80b006c610 con 0x7f80b00b3bd0 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80b70b1640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f80b006c8f0 con 0x7f80b01618a0 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80b70b1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f80b006cbd0 con 0x7f80b00b47b0 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80b4e4d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b00b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80affff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f80b01618a0 0x7f80b0161cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80b564e640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f80b00b3bd0 0x7f80b00b3fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49104/0 (socket says 172.21.15.53:49104) 2024-04-25T19:11:05.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.846+0000 7f80b564e640 1 -- 172.21.15.53:0/519563806 learned_addr learned my addr 172.21.15.53:0/519563806 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-04-25T19:11:05.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80b4e4d640 1 -- 172.21.15.53:0/519563806 >> v2:172.21.15.53:3300/0 conn(0x7f80b01618a0 msgr2=0x7f80b0161cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:05.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80b4e4d640 1 --2- 172.21.15.53:0/519563806 >> v2:172.21.15.53:3300/0 conn(0x7f80b01618a0 0x7f80b0161cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80b4e4d640 1 -- 172.21.15.53:0/519563806 >> v1:172.21.15.53:6789/0 conn(0x7f80b00b3bd0 legacy=0x7f80b00b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80b4e4d640 1 -- 172.21.15.53:0/519563806 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b006ceb0 con 0x7f80b00b47b0 2024-04-25T19:11:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80af7fe640 1 -- 172.21.15.53:0/519563806 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 903776180 0 0) 0x7f80b006c610 con 0x7f80b00b3bd0 2024-04-25T19:11:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80affff640 1 --2- 172.21.15.53:0/519563806 >> v2:172.21.15.53:3300/0 conn(0x7f80b01618a0 0x7f80b0161cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80b4e4d640 1 --2- 172.21.15.53:0/519563806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b00b2370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f80a40097a0 tx=0x7f80a402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2690aeaf2f36b68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80af7fe640 1 -- 172.21.15.53:0/519563806 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a4037070 con 0x7f80b00b47b0 2024-04-25T19:11:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.847+0000 7f80af7fe640 1 -- 172.21.15.53:0/519563806 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a402ae50 con 0x7f80b00b47b0 2024-04-25T19:11:05.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80af7fe640 1 -- 172.21.15.53:0/519563806 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a4032660 con 0x7f80b00b47b0 2024-04-25T19:11:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80b70b1640 1 -- 172.21.15.53:0/519563806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 msgr2=0x7f80b00b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80b70b1640 1 --2- 172.21.15.53:0/519563806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b00b2370 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f80a40097a0 tx=0x7f80a402a100 comp rx=0 tx=0).stop 2024-04-25T19:11:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80b70b1640 1 -- 172.21.15.53:0/519563806 shutdown_connections 2024-04-25T19:11:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80b70b1640 1 --2- 172.21.15.53:0/519563806 >> v2:172.21.15.53:3300/0 conn(0x7f80b01618a0 0x7f80b0161cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80b70b1640 1 --2- 172.21.15.53:0/519563806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b00b2370 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80b70b1640 1 -- 172.21.15.53:0/519563806 >> 172.21.15.53:0/519563806 conn(0x7f80b0073d80 msgr2=0x7f80b009b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80b70b1640 1 -- 172.21.15.53:0/519563806 shutdown_connections 2024-04-25T19:11:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.848+0000 7f80b70b1640 1 -- 172.21.15.53:0/519563806 wait complete. 2024-04-25T19:11:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.850+0000 7f80b70b1640 1 Processor -- start 2024-04-25T19:11:05.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.850+0000 7f80b70b1640 1 -- start start 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80b70b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b00b3bd0 0x7f80b0157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80b70b1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b01f8c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80b70b1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b01618a0 0x7f80b01f9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80b70b1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f80b0161420 con 0x7f80b00b3bd0 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80b70b1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f80b01612a0 con 0x7f80b01618a0 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80b70b1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f80b01615a0 con 0x7f80b00b47b0 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b01f8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b01f8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51606/0 (socket says 172.21.15.53:51606) 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 -- 172.21.15.53:0/977176420 learned_addr learned my addr 172.21.15.53:0/977176420 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:05.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80b564e640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b01618a0 0x7f80b01f9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80b4e4d640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b00b3bd0 0x7f80b0157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 -- 172.21.15.53:0/977176420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b01618a0 msgr2=0x7f80b01f9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b01618a0 0x7f80b01f9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 -- 172.21.15.53:0/977176420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b00b3bd0 msgr2=0x7f80b0157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b00b3bd0 0x7f80b0157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 -- 172.21.15.53:0/977176420 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b015d0c0 con 0x7f80b00b47b0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.852+0000 7f80affff640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b01f8c50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f809c00e870 tx=0x7f809c00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.853+0000 7f80ad7fa640 1 -- 172.21.15.53:0/977176420 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c017070 con 0x7f80b00b47b0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.853+0000 7f80ad7fa640 1 -- 172.21.15.53:0/977176420 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f809c00c940 con 0x7f80b00b47b0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.853+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b015d3a0 con 0x7f80b00b47b0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.853+0000 7f80ad7fa640 1 -- 172.21.15.53:0/977176420 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809c00cd10 con 0x7f80b00b47b0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.853+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b015d960 con 0x7f80b00b47b0 2024-04-25T19:11:05.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.853+0000 7f80ad7fa640 1 -- 172.21.15.53:0/977176420 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f809c01a070 con 0x7f80b00b47b0 2024-04-25T19:11:05.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.853+0000 7f80ad7fa640 1 -- 172.21.15.53:0/977176420 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f809c01c8e0 con 0x7f80b00b47b0 2024-04-25T19:11:05.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.854+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80b0050740 con 0x7f80b00b47b0 2024-04-25T19:11:05.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.858+0000 7f80ad7fa640 1 -- 172.21.15.53:0/977176420 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f809c0092e0 con 0x7f80b00b47b0 2024-04-25T19:11:05.924 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:05.922+0000 7f69f4172040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:11:05.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.932+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd new", "uuid": "420b3523-29ab-4043-bc1b-2457f61a5e04", "id": 3} v 0) -- 0x7f80b0155100 con 0x7f80b00b47b0 2024-04-25T19:11:05.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.936+0000 7f80ad7fa640 1 -- 172.21.15.53:0/977176420 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "420b3523-29ab-4043-bc1b-2457f61a5e04", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f809c01cbe0 con 0x7f80b00b47b0 2024-04-25T19:11:05.938 INFO:teuthology.orchestra.run.smithi053.stdout:3 2024-04-25T19:11:05.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 msgr2=0x7f80b01f8c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:05.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b01f8c50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f809c00e870 tx=0x7f809c00ed30 comp rx=0 tx=0).stop 2024-04-25T19:11:05.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 shutdown_connections 2024-04-25T19:11:05.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b01618a0 0x7f80b01f9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f80b00b47b0 0x7f80b01f8c50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 --2- 172.21.15.53:0/977176420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b00b3bd0 0x7f80b0157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:05.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 >> 172.21.15.53:0/977176420 conn(0x7f80b0073d80 msgr2=0x7f80b00b1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:05.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 shutdown_connections 2024-04-25T19:11:05.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:05.937+0000 7f80b70b1640 1 -- 172.21.15.53:0/977176420 wait complete. 2024-04-25T19:11:05.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new fceab193-1664-4927-9f5e-b04ae6651271 4 2024-04-25T19:11:05.994 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:05.992+0000 7f294431c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T19:11:06.016 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:06.015+0000 7f69f4172040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:11:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.058+0000 7f5448f08640 1 Processor -- start 2024-04-25T19:11:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.058+0000 7f5448f08640 1 -- start start 2024-04-25T19:11:06.061 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:06.059+0000 7f69f4172040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:11:06.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.059+0000 7f5448f08640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f544406dd90 0x7f5444167b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.059+0000 7f5448f08640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54440cf5e0 con 0x7f544415cf20 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.059+0000 7f5448f08640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54440cf8c0 con 0x7f544406e8a0 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.059+0000 7f5448f08640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f54440d05d0 con 0x7f544406dd90 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f544259c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f544406e8a0 0x7f54440b4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49108/0 (socket says 172.21.15.53:49108) 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f5442d9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f544406dd90 0x7f5444167b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f544259c640 1 -- 172.21.15.53:0/1949958254 learned_addr learned my addr 172.21.15.53:0/1949958254 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f5441d9b640 1 -- 172.21.15.53:0/1949958254 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 999874854 0 0) 0x7f54440cf8c0 con 0x7f544406e8a0 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f5441d9b640 1 -- 172.21.15.53:0/1949958254 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5430003520 con 0x7f544406e8a0 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f5442d9d640 1 -- 172.21.15.53:0/1949958254 >> v1:172.21.15.53:6789/0 conn(0x7f544406e8a0 legacy=0x7f54440b4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f5442d9d640 1 -- 172.21.15.53:0/1949958254 >> v1:172.21.15.28:6789/0 conn(0x7f544415cf20 legacy=0x7f544406d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f5441d9b640 1 -- 172.21.15.53:0/1949958254 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2785891148 0 0) 0x7f54440cf5e0 con 0x7f544415cf20 2024-04-25T19:11:06.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.060+0000 7f5442d9d640 1 -- 172.21.15.53:0/1949958254 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54440d08b0 con 0x7f544406dd90 2024-04-25T19:11:06.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.061+0000 7f5442d9d640 1 --2- 172.21.15.53:0/1949958254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544406dd90 0x7f5444167b50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f543c009d90 tx=0x7f543c02c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54ad6fedba6a8241 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:06.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.061+0000 7f5441d9b640 1 -- 172.21.15.53:0/1949958254 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c032770 con 0x7f544406dd90 2024-04-25T19:11:06.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.061+0000 7f5441d9b640 1 -- 172.21.15.53:0/1949958254 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f543c038070 con 0x7f544406dd90 2024-04-25T19:11:06.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.061+0000 7f5441d9b640 1 -- 172.21.15.53:0/1949958254 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c032b30 con 0x7f544406dd90 2024-04-25T19:11:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.062+0000 7f5448f08640 1 -- 172.21.15.53:0/1949958254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544406dd90 msgr2=0x7f5444167b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.062+0000 7f5448f08640 1 --2- 172.21.15.53:0/1949958254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544406dd90 0x7f5444167b50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f543c009d90 tx=0x7f543c02c870 comp rx=0 tx=0).stop 2024-04-25T19:11:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.062+0000 7f5448f08640 1 -- 172.21.15.53:0/1949958254 shutdown_connections 2024-04-25T19:11:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.062+0000 7f5448f08640 1 --2- 172.21.15.53:0/1949958254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544406dd90 0x7f5444167b50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.062+0000 7f5448f08640 1 -- 172.21.15.53:0/1949958254 >> 172.21.15.53:0/1949958254 conn(0x7f54440b7570 msgr2=0x7f54440b7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.062+0000 7f5448f08640 1 -- 172.21.15.53:0/1949958254 shutdown_connections 2024-04-25T19:11:06.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.062+0000 7f5448f08640 1 -- 172.21.15.53:0/1949958254 wait complete. 2024-04-25T19:11:06.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.064+0000 7f5448f08640 1 Processor -- start 2024-04-25T19:11:06.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.064+0000 7f5448f08640 1 -- start start 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5448f08640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544406e8a0 0x7f5444153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5448f08640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544415cf20 0x7f54440b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5448f08640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f54441579d0 0x7f54440d27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5448f08640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54440d1f40 con 0x7f544406e8a0 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5448f08640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f54440d1dc0 con 0x7f544415cf20 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5448f08640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f54440d20c0 con 0x7f54441579d0 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5439d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544415cf20 0x7f54440b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5442d9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f54441579d0 0x7f54440d27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5439d9b640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544415cf20 0x7f54440b00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38570/0 (socket says 172.21.15.53:38570) 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f5439d9b640 1 -- 172.21.15.53:0/694209128 learned_addr learned my addr 172.21.15.53:0/694209128 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.065+0000 7f544259c640 1 --2- 172.21.15.53:0/694209128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544406e8a0 0x7f5444153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f5439d9b640 1 -- 172.21.15.53:0/694209128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f54441579d0 msgr2=0x7f54440d27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f5439d9b640 1 --2- 172.21.15.53:0/694209128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f54441579d0 0x7f54440d27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f5439d9b640 1 -- 172.21.15.53:0/694209128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544406e8a0 msgr2=0x7f5444153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f5439d9b640 1 --2- 172.21.15.53:0/694209128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544406e8a0 0x7f5444153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f5439d9b640 1 -- 172.21.15.53:0/694209128 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5444156c10 con 0x7f544415cf20 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f5439d9b640 1 --2- 172.21.15.53:0/694209128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544415cf20 0x7f54440b00a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f542c00c3e0 tx=0x7f542c00c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f543b7fe640 1 -- 172.21.15.53:0/694209128 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c017070 con 0x7f544415cf20 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f543b7fe640 1 -- 172.21.15.53:0/694209128 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f542c0035f0 con 0x7f544415cf20 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5444156ef0 con 0x7f544415cf20 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f543b7fe640 1 -- 172.21.15.53:0/694209128 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f542c0039c0 con 0x7f544415cf20 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54440b1690 con 0x7f544415cf20 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f543b7fe640 1 -- 172.21.15.53:0/694209128 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f542c01a070 con 0x7f544415cf20 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.066+0000 7f543b7fe640 1 -- 172.21.15.53:0/694209128 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f542c012910 con 0x7f544415cf20 2024-04-25T19:11:06.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.067+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5444050740 con 0x7f544415cf20 2024-04-25T19:11:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.069+0000 7f543b7fe640 1 -- 172.21.15.53:0/694209128 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f542c021070 con 0x7f544415cf20 2024-04-25T19:11:06.089 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:06.088+0000 7f294431c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T19:11:06.135 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:06.133+0000 7f294431c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T19:11:06.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.143+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fceab193-1664-4927-9f5e-b04ae6651271", "id": 4} v 0) -- 0x7f544415f5f0 con 0x7f544415cf20 2024-04-25T19:11:06.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.147+0000 7f543b7fe640 1 -- 172.21.15.53:0/694209128 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fceab193-1664-4927-9f5e-b04ae6651271", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f542c003d60 con 0x7f544415cf20 2024-04-25T19:11:06.148 INFO:teuthology.orchestra.run.smithi053.stdout:4 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544415cf20 msgr2=0x7f54440b00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 --2- 172.21.15.53:0/694209128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544415cf20 0x7f54440b00a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f542c00c3e0 tx=0x7f542c00c8a0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 shutdown_connections 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 --2- 172.21.15.53:0/694209128 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f54441579d0 0x7f54440d27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 --2- 172.21.15.53:0/694209128 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f544415cf20 0x7f54440b00a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 --2- 172.21.15.53:0/694209128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544406e8a0 0x7f5444153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 >> 172.21.15.53:0/694209128 conn(0x7f54440b7570 msgr2=0x7f544406c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 shutdown_connections 2024-04-25T19:11:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.148+0000 7f5448f08640 1 -- 172.21.15.53:0/694209128 wait complete. 2024-04-25T19:11:06.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 041516ce-9cb0-49e2-9993-5a3968f84727 5 2024-04-25T19:11:06.236 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:06.234+0000 7f69f4172040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:11:06.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.270+0000 7f84380fa640 1 Processor -- start 2024-04-25T19:11:06.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.270+0000 7f84380fa640 1 -- start start 2024-04-25T19:11:06.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f84380fa640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f843006edb0 0x7f84300b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f84380fa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84301616a0 0x7f843009b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f84380fa640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f84300d1b30 0x7f84300d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f84380fa640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f843009c190 con 0x7f843006edb0 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f84380fa640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f8430160c00 con 0x7f84300d1b30 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f84380fa640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f843009beb0 con 0x7f84301616a0 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f8435695640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84301616a0 0x7f843009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f8436697640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f84300d1b30 0x7f84300d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f8435695640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84301616a0 0x7f843009b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51622/0 (socket says 172.21.15.53:51622) 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f8435695640 1 -- 172.21.15.53:0/4293077428 learned_addr learned my addr 172.21.15.53:0/4293077428 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f8436697640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f84300d1b30 0x7f84300d1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38586/0 (socket says 172.21.15.53:38586) 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.271+0000 7f8435e96640 1 --2- 172.21.15.53:0/4293077428 >> v2:172.21.15.28:3300/0 conn(0x7f843006edb0 0x7f84300b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.272+0000 7f8435695640 1 -- 172.21.15.53:0/4293077428 >> v2:172.21.15.53:3300/0 conn(0x7f84300d1b30 msgr2=0x7f84300d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:06.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.272+0000 7f8435695640 1 --2- 172.21.15.53:0/4293077428 >> v2:172.21.15.53:3300/0 conn(0x7f84300d1b30 0x7f84300d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.272+0000 7f8435695640 1 -- 172.21.15.53:0/4293077428 >> v2:172.21.15.28:3300/0 conn(0x7f843006edb0 msgr2=0x7f84300b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:06.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.272+0000 7f8435695640 1 --2- 172.21.15.53:0/4293077428 >> v2:172.21.15.28:3300/0 conn(0x7f843006edb0 0x7f84300b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.272+0000 7f8435695640 1 -- 172.21.15.53:0/4293077428 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8430160ea0 con 0x7f84301616a0 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.272+0000 7f8435695640 1 --2- 172.21.15.53:0/4293077428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84301616a0 0x7f843009b540 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f84300b88d0 tx=0x7f842402a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a819e7e7d93ef80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f8434e94640 1 -- 172.21.15.53:0/4293077428 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424037070 con 0x7f84301616a0 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f8434e94640 1 -- 172.21.15.53:0/4293077428 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8424032440 con 0x7f84301616a0 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f8434e94640 1 -- 172.21.15.53:0/4293077428 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8424032810 con 0x7f84301616a0 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f84380fa640 1 -- 172.21.15.53:0/4293077428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84301616a0 msgr2=0x7f843009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f84380fa640 1 --2- 172.21.15.53:0/4293077428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84301616a0 0x7f843009b540 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f84300b88d0 tx=0x7f842402a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f84380fa640 1 -- 172.21.15.53:0/4293077428 shutdown_connections 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f84380fa640 1 --2- 172.21.15.53:0/4293077428 >> v2:172.21.15.53:3300/0 conn(0x7f84300d1b30 0x7f84300d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f84380fa640 1 --2- 172.21.15.53:0/4293077428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84301616a0 0x7f843009b540 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f84380fa640 1 --2- 172.21.15.53:0/4293077428 >> v2:172.21.15.28:3300/0 conn(0x7f843006edb0 0x7f84300b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.273+0000 7f84380fa640 1 -- 172.21.15.53:0/4293077428 >> 172.21.15.53:0/4293077428 conn(0x7f84300b7620 msgr2=0x7f84300b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:06.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.274+0000 7f84380fa640 1 -- 172.21.15.53:0/4293077428 shutdown_connections 2024-04-25T19:11:06.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.274+0000 7f84380fa640 1 -- 172.21.15.53:0/4293077428 wait complete. 2024-04-25T19:11:06.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.275+0000 7f84380fa640 1 Processor -- start 2024-04-25T19:11:06.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.275+0000 7f84380fa640 1 -- start start 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84380fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843006edb0 0x7f843014f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84380fa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84300d1b30 0x7f8430156930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84380fa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84301616a0 0x7f8430156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84380fa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84300d2480 con 0x7f843006edb0 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84380fa640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f84300b4180 con 0x7f84301616a0 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84380fa640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f843009ba80 con 0x7f84300d1b30 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8435695640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84300d1b30 0x7f8430156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84301616a0 0x7f8430156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84301616a0 0x7f8430156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38596/0 (socket says 172.21.15.53:38596) 2024-04-25T19:11:06.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8435695640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84300d1b30 0x7f8430156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51630/0 (socket says 172.21.15.53:51630) 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 -- 172.21.15.53:0/2463855200 learned_addr learned my addr 172.21.15.53:0/2463855200 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8435e96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843006edb0 0x7f843014f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 -- 172.21.15.53:0/2463855200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84300d1b30 msgr2=0x7f8430156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84300d1b30 0x7f8430156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 -- 172.21.15.53:0/2463855200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843006edb0 msgr2=0x7f843014f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843006edb0 0x7f843014f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 -- 172.21.15.53:0/2463855200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8430161c30 con 0x7f84301616a0 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8435e96640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843006edb0 0x7f843014f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8435695640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84300d1b30 0x7f8430156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f8436697640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84301616a0 0x7f8430156e70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8428009d30 tx=0x7f842800c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:06.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84227fc640 1 -- 172.21.15.53:0/2463855200 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428018020 con 0x7f84301616a0 2024-04-25T19:11:06.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84227fc640 1 -- 172.21.15.53:0/2463855200 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8428014480 con 0x7f84301616a0 2024-04-25T19:11:06.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f843006d3a0 con 0x7f84301616a0 2024-04-25T19:11:06.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.277+0000 7f84227fc640 1 -- 172.21.15.53:0/2463855200 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428014870 con 0x7f84301616a0 2024-04-25T19:11:06.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.278+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f843006d960 con 0x7f84301616a0 2024-04-25T19:11:06.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.278+0000 7f84227fc640 1 -- 172.21.15.53:0/2463855200 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8428005d10 con 0x7f84301616a0 2024-04-25T19:11:06.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.278+0000 7f84227fc640 1 -- 172.21.15.53:0/2463855200 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f8428013960 con 0x7f84301616a0 2024-04-25T19:11:06.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.278+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8430161e00 con 0x7f84301616a0 2024-04-25T19:11:06.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.281+0000 7f84227fc640 1 -- 172.21.15.53:0/2463855200 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8428022070 con 0x7f84301616a0 2024-04-25T19:11:06.294 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:06.293+0000 7f69f4172040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:11:06.323 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:06.322+0000 7f294431c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T19:11:06.349 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:06.348+0000 7f69f4172040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:11:06.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.355+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd new", "uuid": "041516ce-9cb0-49e2-9993-5a3968f84727", "id": 5} v 0) -- 0x7f8430050740 con 0x7f84301616a0 2024-04-25T19:11:06.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.359+0000 7f84227fc640 1 -- 172.21.15.53:0/2463855200 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "041516ce-9cb0-49e2-9993-5a3968f84727", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f8428010040 con 0x7f84301616a0 2024-04-25T19:11:06.361 INFO:teuthology.orchestra.run.smithi053.stdout:5 2024-04-25T19:11:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84301616a0 msgr2=0x7f8430156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84301616a0 0x7f8430156e70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8428009d30 tx=0x7f842800c490 comp rx=0 tx=0).stop 2024-04-25T19:11:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 shutdown_connections 2024-04-25T19:11:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84301616a0 0x7f8430156e70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84300d1b30 0x7f8430156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 --2- 172.21.15.53:0/2463855200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f843006edb0 0x7f843014f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 >> 172.21.15.53:0/2463855200 conn(0x7f84300b7620 msgr2=0x7f843015a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 shutdown_connections 2024-04-25T19:11:06.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.360+0000 7f84380fa640 1 -- 172.21.15.53:0/2463855200 wait complete. 2024-04-25T19:11:06.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new cb5e7877-ce29-42c3-9557-3e1f3cc2395f 6 2024-04-25T19:11:06.380 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:06.379+0000 7f294431c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T19:11:06.437 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:06.436+0000 7f294431c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T19:11:06.469 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:06.466+0000 7f69f4172040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:11:06.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.482+0000 7f0de3284640 1 Processor -- start 2024-04-25T19:11:06.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.482+0000 7f0de3284640 1 -- start start 2024-04-25T19:11:06.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de3284640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0ddc15e220 0x7f0ddc157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de3284640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc09b540 0x7f0ddc09b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de3284640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ddc15e630 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de3284640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f0ddc09be90 con 0x7f0ddc15e220 2024-04-25T19:11:06.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de3284640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc0cffe0 con 0x7f0ddc09b540 2024-04-25T19:11:06.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de081f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0ddc15e220 0x7f0ddc157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de1821640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc09b540 0x7f0ddc09b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de081f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0ddc15e220 0x7f0ddc157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38600/0 (socket says 172.21.15.53:38600) 2024-04-25T19:11:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de1020640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0ddc0cfa00 0x7f0ddc0b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49114/0 (socket says 172.21.15.53:49114) 2024-04-25T19:11:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de1821640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc09b540 0x7f0ddc09b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51646/0 (socket says 172.21.15.53:51646) 2024-04-25T19:11:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.483+0000 7f0de081f640 1 -- 172.21.15.53:0/1858687930 learned_addr learned my addr 172.21.15.53:0/1858687930 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.484+0000 7f0de081f640 1 -- 172.21.15.53:0/1858687930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc09b540 msgr2=0x7f0ddc09b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.484+0000 7f0de081f640 1 --2- 172.21.15.53:0/1858687930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc09b540 0x7f0ddc09b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.484+0000 7f0de081f640 1 -- 172.21.15.53:0/1858687930 >> v1:172.21.15.53:6789/0 conn(0x7f0ddc0cfa00 legacy=0x7f0ddc0b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.484+0000 7f0de081f640 1 -- 172.21.15.53:0/1858687930 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ddc0d02c0 con 0x7f0ddc15e220 2024-04-25T19:11:06.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.484+0000 7f0de1821640 1 --2- 172.21.15.53:0/1858687930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc09b540 0x7f0ddc09b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:06.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.484+0000 7f0de081f640 1 --2- 172.21.15.53:0/1858687930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc157570 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8009870 tx=0x7f0dd802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ced4e7df3c35881b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:06.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0dd3fff640 1 -- 172.21.15.53:0/1858687930 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8037070 con 0x7f0ddc15e220 2024-04-25T19:11:06.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0dd3fff640 1 -- 172.21.15.53:0/1858687930 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd8032440 con 0x7f0ddc15e220 2024-04-25T19:11:06.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0dd3fff640 1 -- 172.21.15.53:0/1858687930 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8032810 con 0x7f0ddc15e220 2024-04-25T19:11:06.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0de3284640 1 -- 172.21.15.53:0/1858687930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 msgr2=0x7f0ddc157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0de3284640 1 --2- 172.21.15.53:0/1858687930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc157570 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8009870 tx=0x7f0dd802a1c0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0de3284640 1 -- 172.21.15.53:0/1858687930 shutdown_connections 2024-04-25T19:11:06.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0de3284640 1 --2- 172.21.15.53:0/1858687930 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc09b540 0x7f0ddc09b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0de3284640 1 --2- 172.21.15.53:0/1858687930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc157570 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.485+0000 7f0de3284640 1 -- 172.21.15.53:0/1858687930 >> 172.21.15.53:0/1858687930 conn(0x7f0ddc0b7620 msgr2=0x7f0ddc0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:06.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.486+0000 7f0de3284640 1 -- 172.21.15.53:0/1858687930 shutdown_connections 2024-04-25T19:11:06.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.486+0000 7f0de3284640 1 -- 172.21.15.53:0/1858687930 wait complete. 2024-04-25T19:11:06.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.487+0000 7f0de3284640 1 Processor -- start 2024-04-25T19:11:06.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.487+0000 7f0de3284640 1 -- start start 2024-04-25T19:11:06.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de3284640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ddc09b540 0x7f0ddc150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de3284640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc0cfa00 0x7f0ddc0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de3284640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc15c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de3284640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc15d680 con 0x7f0ddc09b540 2024-04-25T19:11:06.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de3284640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0ddc157ab0 con 0x7f0ddc15e220 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de3284640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0ddc15d800 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc0cfa00 0x7f0ddc0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de1821640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc15c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc0cfa00 0x7f0ddc0b1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51656/0 (socket says 172.21.15.53:51656) 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de1821640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc15c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38610/0 (socket says 172.21.15.53:38610) 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 -- 172.21.15.53:0/3901985864 learned_addr learned my addr 172.21.15.53:0/3901985864 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de1020640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ddc09b540 0x7f0ddc150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 -- 172.21.15.53:0/3901985864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 msgr2=0x7f0ddc15c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc15c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 -- 172.21.15.53:0/3901985864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ddc09b540 msgr2=0x7f0ddc150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ddc09b540 0x7f0ddc150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 -- 172.21.15.53:0/3901985864 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ddc0b0b20 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de1821640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:06.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0de081f640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc0cfa00 0x7f0ddc0b1f00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80090b0 tx=0x7f0dd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:06.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.489+0000 7f0dd1ffb640 1 -- 172.21.15.53:0/3901985864 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8041030 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.490+0000 7f0dd1ffb640 1 -- 172.21.15.53:0/3901985864 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd8030440 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.490+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ddc0b0e00 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.490+0000 7f0dd1ffb640 1 -- 172.21.15.53:0/3901985864 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8037040 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.490+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ddc0b13c0 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.490+0000 7f0dd1ffb640 1 -- 172.21.15.53:0/3901985864 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0dd8044070 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.490+0000 7f0dd1ffb640 1 -- 172.21.15.53:0/3901985864 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f0dd803db50 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.491+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ddc0b1600 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.493+0000 7f0dd1ffb640 1 -- 172.21.15.53:0/3901985864 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0ddc0b1600 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.559 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:06.557+0000 7f294431c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T19:11:06.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.568+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cb5e7877-ce29-42c3-9557-3e1f3cc2395f", "id": 6} v 0) -- 0x7f0ddc050740 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.571+0000 7f0dd1ffb640 1 -- 172.21.15.53:0/3901985864 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb5e7877-ce29-42c3-9557-3e1f3cc2395f", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f0dd803ddf0 con 0x7f0ddc0cfa00 2024-04-25T19:11:06.573 INFO:teuthology.orchestra.run.smithi053.stdout:6 2024-04-25T19:11:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc0cfa00 msgr2=0x7f0ddc0b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc0cfa00 0x7f0ddc0b1f00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0dd80090b0 tx=0x7f0dd80046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 shutdown_connections 2024-04-25T19:11:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc15e220 0x7f0ddc15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ddc0cfa00 0x7f0ddc0b1f00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 --2- 172.21.15.53:0/3901985864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ddc09b540 0x7f0ddc150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 >> 172.21.15.53:0/3901985864 conn(0x7f0ddc0b7620 msgr2=0x7f0ddc070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:06.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 shutdown_connections 2024-04-25T19:11:06.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.572+0000 7f0de3284640 1 -- 172.21.15.53:0/3901985864 wait complete. 2024-04-25T19:11:06.585 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd new 07a614e4-04f7-4d78-a275-f4a9c3267e8e 7 2024-04-25T19:11:06.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.735+0000 7f4259261640 1 Processor -- start 2024-04-25T19:11:06.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.735+0000 7f4259261640 1 -- start start 2024-04-25T19:11:06.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.736+0000 7f4259261640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42540b4ad0 0x7f42540b3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.736+0000 7f4259261640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42540b20c0 con 0x7f42540b3810 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.736+0000 7f4259261640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42540b2360 con 0x7f42540b3100 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.736+0000 7f4259261640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f42540b2c70 con 0x7f42540b4ad0 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.736+0000 7f42537fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42540b4ad0 0x7f42540b3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.736+0000 7f4252ffd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f42540b3100 0x7f42540cfcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49118/0 (socket says 172.21.15.53:49118) 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.736+0000 7f42537fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42540b4ad0 0x7f42540b3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38612/0 (socket says 172.21.15.53:38612) 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.736+0000 7f4252ffd640 1 -- 172.21.15.53:0/1505940823 learned_addr learned my addr 172.21.15.53:0/1505940823 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.737+0000 7f42537fe640 1 -- 172.21.15.53:0/1505940823 >> v1:172.21.15.53:6789/0 conn(0x7f42540b3100 legacy=0x7f42540cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.737+0000 7f42537fe640 1 -- 172.21.15.53:0/1505940823 >> v1:172.21.15.28:6789/0 conn(0x7f42540b3810 legacy=0x7f42540b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.737+0000 7f42537fe640 1 -- 172.21.15.53:0/1505940823 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42540b2620 con 0x7f42540b4ad0 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.737+0000 7f42537fe640 1 --2- 172.21.15.53:0/1505940823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b4ad0 0x7f42540b3c20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f42540b2df0 tx=0x7f424c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8199eb0fb9c93176 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.737+0000 7f42527fc640 1 -- 172.21.15.53:0/1505940823 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c007740 con 0x7f42540b4ad0 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.737+0000 7f42527fc640 1 -- 172.21.15.53:0/1505940823 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f424c0078e0 con 0x7f42540b4ad0 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.737+0000 7f42527fc640 1 -- 172.21.15.53:0/1505940823 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f424c007cb0 con 0x7f42540b4ad0 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.738+0000 7f4259261640 1 -- 172.21.15.53:0/1505940823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b4ad0 msgr2=0x7f42540b3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.738+0000 7f4259261640 1 --2- 172.21.15.53:0/1505940823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b4ad0 0x7f42540b3c20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f42540b2df0 tx=0x7f424c02a6f0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.738+0000 7f4259261640 1 -- 172.21.15.53:0/1505940823 shutdown_connections 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.738+0000 7f4259261640 1 --2- 172.21.15.53:0/1505940823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b4ad0 0x7f42540b3c20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.738+0000 7f4259261640 1 -- 172.21.15.53:0/1505940823 >> 172.21.15.53:0/1505940823 conn(0x7f425409b540 msgr2=0x7f425409b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.738+0000 7f4259261640 1 -- 172.21.15.53:0/1505940823 shutdown_connections 2024-04-25T19:11:06.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.738+0000 7f4259261640 1 -- 172.21.15.53:0/1505940823 wait complete. 2024-04-25T19:11:06.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.740+0000 7f4259261640 1 Processor -- start 2024-04-25T19:11:06.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.740+0000 7f4259261640 1 -- start start 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.741+0000 7f4259261640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b3100 0x7f4254155150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.741+0000 7f4259261640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42540b3810 0x7f4254155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.741+0000 7f4259261640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254156190 0x7f425409e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.741+0000 7f4259261640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4254169140 con 0x7f4254156190 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.741+0000 7f4259261640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4254168fc0 con 0x7f42540b3100 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.741+0000 7f4259261640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f42541692c0 con 0x7f42540b3810 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424bfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42540b3810 0x7f4254155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f4252ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b3100 0x7f4254155150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f4252ffd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b3100 0x7f4254155150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38614/0 (socket says 172.21.15.53:38614) 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424bfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42540b3810 0x7f4254155c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.53:51662/0 (socket says 172.21.15.53:51662) 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f4252ffd640 1 -- 172.21.15.53:0/410371233 learned_addr learned my addr 172.21.15.53:0/410371233 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f42537fe640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254156190 0x7f425409e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424bfff640 1 -- 172.21.15.53:0/410371233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b3100 msgr2=0x7f4254155150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424bfff640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b3100 0x7f4254155150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424bfff640 1 -- 172.21.15.53:0/410371233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254156190 msgr2=0x7f425409e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424bfff640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254156190 0x7f425409e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424bfff640 1 -- 172.21.15.53:0/410371233 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42541565a0 con 0x7f42540b3810 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f4252ffd640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b3100 0x7f4254155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424bfff640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42540b3810 0x7f4254155c50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4254074130 tx=0x7f423c00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424b7fe640 1 -- 172.21.15.53:0/410371233 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c017070 con 0x7f42540b3810 2024-04-25T19:11:06.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424b7fe640 1 -- 172.21.15.53:0/410371233 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f423c0052d0 con 0x7f42540b3810 2024-04-25T19:11:06.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4254153d30 con 0x7f42540b3810 2024-04-25T19:11:06.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f424b7fe640 1 -- 172.21.15.53:0/410371233 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f423c003850 con 0x7f42540b3810 2024-04-25T19:11:06.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.742+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f42541542b0 con 0x7f42540b3810 2024-04-25T19:11:06.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.743+0000 7f424b7fe640 1 -- 172.21.15.53:0/410371233 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f423c01a070 con 0x7f42540b3810 2024-04-25T19:11:06.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.743+0000 7f424b7fe640 1 -- 172.21.15.53:0/410371233 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f423c011590 con 0x7f42540b3810 2024-04-25T19:11:06.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.743+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42540cf650 con 0x7f42540b3810 2024-04-25T19:11:06.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.746+0000 7f424b7fe640 1 -- 172.21.15.53:0/410371233 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f42540cf650 con 0x7f42540b3810 2024-04-25T19:11:06.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.820+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd new", "uuid": "07a614e4-04f7-4d78-a275-f4a9c3267e8e", "id": 7} v 0) -- 0x7f4254050740 con 0x7f42540b3810 2024-04-25T19:11:06.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.823+0000 7f424b7fe640 1 -- 172.21.15.53:0/410371233 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "07a614e4-04f7-4d78-a275-f4a9c3267e8e", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f42541565a0 con 0x7f42540b3810 2024-04-25T19:11:06.825 INFO:teuthology.orchestra.run.smithi053.stdout:7 2024-04-25T19:11:06.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.824+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42540b3810 msgr2=0x7f4254155c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.824+0000 7f4259261640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42540b3810 0x7f4254155c50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4254074130 tx=0x7f423c00c820 comp rx=0 tx=0).stop 2024-04-25T19:11:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.825+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 shutdown_connections 2024-04-25T19:11:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.825+0000 7f4259261640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4254156190 0x7f425409e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.825+0000 7f4259261640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42540b3810 0x7f4254155c50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.825+0000 7f4259261640 1 --2- 172.21.15.53:0/410371233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42540b3100 0x7f4254155150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.825+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 >> 172.21.15.53:0/410371233 conn(0x7f425409b540 msgr2=0x7f4254161bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.825+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 shutdown_connections 2024-04-25T19:11:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:11:06.825+0000 7f4259261640 1 -- 172.21.15.53:0/410371233 wait complete. 2024-04-25T19:11:06.837 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T19:11:06.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T19:11:06.840 INFO:tasks.ceph.osd.0:Started 2024-04-25T19:11:06.840 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T19:11:06.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T19:11:06.844 INFO:tasks.ceph.osd.1:Started 2024-04-25T19:11:06.844 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T19:11:06.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T19:11:06.846 INFO:tasks.ceph.osd.2:Started 2024-04-25T19:11:06.846 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T19:11:06.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T19:11:06.849 INFO:tasks.ceph.osd.3:Started 2024-04-25T19:11:06.849 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T19:11:06.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T19:11:06.851 INFO:tasks.ceph.osd.4:Started 2024-04-25T19:11:06.851 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T19:11:06.851 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T19:11:06.853 INFO:tasks.ceph.osd.5:Started 2024-04-25T19:11:06.853 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T19:11:06.854 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T19:11:06.856 INFO:tasks.ceph.osd.6:Started 2024-04-25T19:11:06.856 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T19:11:06.856 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T19:11:06.858 INFO:tasks.ceph.osd.7:Started 2024-04-25T19:11:06.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:06.884 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:06.882+0000 7f69f4172040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:11:06.989 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:06.987+0000 7f294431c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T19:11:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:06.998+0000 7fad133b7640 1 Processor -- start 2024-04-25T19:11:07.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:06.998+0000 7fad133b7640 1 -- start start 2024-04-25T19:11:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad133b7640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fad0c0b7d70 0x7fad0c15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad133b7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad133b7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad0c1614a0 con 0x7fad0c164990 2024-04-25T19:11:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad133b7640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fad0c163eb0 con 0x7fad0c0b7d70 2024-04-25T19:11:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad133b7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c164190 con 0x7fad0c1606f0 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad11153640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fad0c164990 0x7fad0c0b7660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54866/0 (socket says 172.21.15.28:54866) 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad11153640 1 -- 172.21.15.28:0/4092886888 learned_addr learned my addr 172.21.15.28:0/4092886888 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad10952640 1 --2- 172.21.15.28:0/4092886888 >> v2:172.21.15.53:3300/0 conn(0x7fad0c0b7d70 0x7fad0c15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad11954640 1 --2- 172.21.15.28:0/4092886888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad03fff640 1 -- 172.21.15.28:0/4092886888 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474972164 0 0) 0x7fad0c1614a0 con 0x7fad0c164990 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad03fff640 1 -- 172.21.15.28:0/4092886888 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facf8003520 con 0x7fad0c164990 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad11954640 1 -- 172.21.15.28:0/4092886888 >> v2:172.21.15.53:3300/0 conn(0x7fad0c0b7d70 msgr2=0x7fad0c15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad11954640 1 --2- 172.21.15.28:0/4092886888 >> v2:172.21.15.53:3300/0 conn(0x7fad0c0b7d70 0x7fad0c15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad11954640 1 -- 172.21.15.28:0/4092886888 >> v1:172.21.15.28:6789/0 conn(0x7fad0c164990 legacy=0x7fad0c0b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.000+0000 7fad11954640 1 -- 172.21.15.28:0/4092886888 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c164470 con 0x7fad0c1606f0 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.001+0000 7fad10952640 1 --2- 172.21.15.28:0/4092886888 >> v2:172.21.15.53:3300/0 conn(0x7fad0c0b7d70 0x7fad0c15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.001+0000 7fad11954640 1 --2- 172.21.15.28:0/4092886888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c160b00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fad04009d90 tx=0x7fad0402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2211713445f8fb46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:07.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.001+0000 7fad03fff640 1 -- 172.21.15.28:0/4092886888 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04007740 con 0x7fad0c1606f0 2024-04-25T19:11:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad03fff640 1 -- 172.21.15.28:0/4092886888 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad040078e0 con 0x7fad0c1606f0 2024-04-25T19:11:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad03fff640 1 -- 172.21.15.28:0/4092886888 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04007cb0 con 0x7fad0c1606f0 2024-04-25T19:11:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad133b7640 1 -- 172.21.15.28:0/4092886888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 msgr2=0x7fad0c160b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:07.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad133b7640 1 --2- 172.21.15.28:0/4092886888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c160b00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fad04009d90 tx=0x7fad0402a660 comp rx=0 tx=0).stop 2024-04-25T19:11:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad133b7640 1 -- 172.21.15.28:0/4092886888 shutdown_connections 2024-04-25T19:11:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad133b7640 1 --2- 172.21.15.28:0/4092886888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c160b00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad133b7640 1 --2- 172.21.15.28:0/4092886888 >> v2:172.21.15.53:3300/0 conn(0x7fad0c0b7d70 0x7fad0c15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad133b7640 1 -- 172.21.15.28:0/4092886888 >> 172.21.15.28:0/4092886888 conn(0x7fad0c0d2840 msgr2=0x7fad0c0d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.002+0000 7fad133b7640 1 -- 172.21.15.28:0/4092886888 shutdown_connections 2024-04-25T19:11:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.003+0000 7fad133b7640 1 -- 172.21.15.28:0/4092886888 wait complete. 2024-04-25T19:11:07.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.004+0000 7fad133b7640 1 Processor -- start 2024-04-25T19:11:07.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.004+0000 7fad133b7640 1 -- start start 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.005+0000 7fad133b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c0b7d70 0x7fad0c0cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad133b7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c0ba3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad133b7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0c164990 0x7fad0c15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad133b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0b4750 con 0x7fad0c0b7d70 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad133b7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0b45d0 con 0x7fad0c164990 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad133b7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c0b48d0 con 0x7fad0c1606f0 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c0b7d70 0x7fad0c0cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c0b7d70 0x7fad0c0cd9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37940/0 (socket says 172.21.15.28:37940) 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 -- 172.21.15.28:0/914462145 learned_addr learned my addr 172.21.15.28:0/914462145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11954640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0c164990 0x7fad0c15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:07.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad10952640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c0ba3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 -- 172.21.15.28:0/914462145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 msgr2=0x7fad0c0ba3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c0ba3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 -- 172.21.15.28:0/914462145 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0c164990 msgr2=0x7fad0c15a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0c164990 0x7fad0c15a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 -- 172.21.15.28:0/914462145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c1f64e0 con 0x7fad0c0b7d70 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad11153640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c0b7d70 0x7fad0c0cd9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c3c0 tx=0x7facfc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad01ffb640 1 -- 172.21.15.28:0/914462145 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc0074e0 con 0x7fad0c0b7d70 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad01ffb640 1 -- 172.21.15.28:0/914462145 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facfc0047e0 con 0x7fad0c0b7d70 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0c0bacc0 con 0x7fad0c0b7d70 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.006+0000 7fad01ffb640 1 -- 172.21.15.28:0/914462145 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc004c50 con 0x7fad0c0b7d70 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.007+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0c0cd1c0 con 0x7fad0c0b7d70 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.007+0000 7fad01ffb640 1 -- 172.21.15.28:0/914462145 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7facfc011070 con 0x7fad0c0b7d70 2024-04-25T19:11:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.007+0000 7fad01ffb640 1 -- 172.21.15.28:0/914462145 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7facfc013a90 con 0x7fad0c0b7d70 2024-04-25T19:11:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.008+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad0c050740 con 0x7fad0c0b7d70 2024-04-25T19:11:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.010+0000 7fad01ffb640 1 -- 172.21.15.28:0/914462145 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7facfc004df0 con 0x7fad0c0b7d70 2024-04-25T19:11:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.073+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad0c0d4000 con 0x7fad0c0b7d70 2024-04-25T19:11:07.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.073+0000 7fad01ffb640 1 -- 172.21.15.28:0/914462145 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7facfc013ce0 con 0x7fad0c0b7d70 2024-04-25T19:11:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:07.075 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:06.822340+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:07.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c0b7d70 msgr2=0x7fad0c0cd9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:07.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c0b7d70 0x7fad0c0cd9c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7facfc00c3c0 tx=0x7facfc00c880 comp rx=0 tx=0).stop 2024-04-25T19:11:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 shutdown_connections 2024-04-25T19:11:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0c164990 0x7fad0c15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad0c1606f0 0x7fad0c0ba3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 --2- 172.21.15.28:0/914462145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad0c0b7d70 0x7fad0c0cd9c0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 >> 172.21.15.28:0/914462145 conn(0x7fad0c0d2840 msgr2=0x7fad0c0d5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 shutdown_connections 2024-04-25T19:11:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:07.074+0000 7fad133b7640 1 -- 172.21.15.28:0/914462145 wait complete. 2024-04-25T19:11:07.087 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T19:11:07.088 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T19:11:07.190 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:07.189+0000 7f69f4172040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:11:07.245 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:07.243+0000 7f69f4172040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:11:07.261 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:07.259+0000 7f294431c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T19:11:07.316 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:07.314+0000 7f294431c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T19:11:07.367 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:07.366+0000 7f69f4172040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:11:07.438 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:07.437+0000 7f294431c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T19:11:07.471 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-04-25T19:11:07.469+0000 7f21317d6640 -1 Falling back to public interface 2024-04-25T19:11:07.473 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-04-25T19:11:07.472+0000 7f77edc2c640 -1 Falling back to public interface 2024-04-25T19:11:07.486 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-04-25T19:11:07.483+0000 7f4c21fcb640 -1 Falling back to public interface 2024-04-25T19:11:07.487 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-04-25T19:11:07.483+0000 7ff1caa21640 -1 Falling back to public interface 2024-04-25T19:11:07.487 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-04-25T19:11:07.484+0000 7f4c20f7a640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2024-04-25T19:11:07.489 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-04-25T19:11:07.487+0000 7fde0521a640 -1 Falling back to public interface 2024-04-25T19:11:07.489 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-04-25T19:11:07.488+0000 7f2928816640 -1 Falling back to public interface 2024-04-25T19:11:07.496 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-04-25T19:11:07.494+0000 7fdff0216640 -1 Falling back to public interface 2024-04-25T19:11:07.496 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-04-25T19:11:07.495+0000 7f0c91611640 -1 Falling back to public interface 2024-04-25T19:11:07.497 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-04-25T19:11:07.495+0000 7f0c8ffb1640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6817/0: (98) Address already in use 2024-04-25T19:11:07.665 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:07.664+0000 7f69f4172040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:11:07.716 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:07.714+0000 7f69f4172040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:11:07.735 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:07.733+0000 7f294431c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T19:11:07.785 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:07.784+0000 7f294431c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T19:11:07.880 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:07.878+0000 7f69f4172040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:11:07.898 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:07.949 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:07.948+0000 7f294431c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T19:11:08.002 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.000+0000 7f60e46d9640 1 Processor -- start 2024-04-25T19:11:08.002 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.000+0000 7f60e46d9640 1 -- start start 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e46d9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 0x7f60dc0d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e46d9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f60dc0d20c0 0x7f60dc0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e46d9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60dc0cd490 0x7f60dc0cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e46d9640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f60dc0ceae0 con 0x7f60dc0cd490 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e46d9640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f60dc1607b0 con 0x7f60dc0d20c0 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e46d9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f60dc0ce240 con 0x7f60dc0cfca0 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e2c76640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60dc0cd490 0x7f60dc0cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e2c76640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60dc0cd490 0x7f60dc0cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37954/0 (socket says 172.21.15.28:37954) 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e2c76640 1 -- 172.21.15.28:0/4289096093 learned_addr learned my addr 172.21.15.28:0/4289096093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e1c74640 1 --2- 172.21.15.28:0/4289096093 >> v2:172.21.15.53:3300/0 conn(0x7f60dc0d20c0 0x7f60dc0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.001+0000 7f60e2475640 1 --2- 172.21.15.28:0/4289096093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 0x7f60dc0d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.002+0000 7f60e2c76640 1 -- 172.21.15.28:0/4289096093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 msgr2=0x7f60dc0d1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:08.004 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.002+0000 7f60e2c76640 1 --2- 172.21.15.28:0/4289096093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 0x7f60dc0d1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.005 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.002+0000 7f60e2c76640 1 -- 172.21.15.28:0/4289096093 >> v2:172.21.15.53:3300/0 conn(0x7f60dc0d20c0 msgr2=0x7f60dc0ccf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:08.005 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.002+0000 7f60e2c76640 1 --2- 172.21.15.28:0/4289096093 >> v2:172.21.15.53:3300/0 conn(0x7f60dc0d20c0 0x7f60dc0ccf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.005 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.002+0000 7f60e2c76640 1 -- 172.21.15.28:0/4289096093 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc0ce520 con 0x7f60dc0cd490 2024-04-25T19:11:08.006 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.002+0000 7f60e2c76640 1 --2- 172.21.15.28:0/4289096093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0cd490 0x7f60dc0cd8a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0b8820 tx=0x7f60d402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88cd3a1f101301d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:08.006 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.003+0000 7f60e1473640 1 -- 172.21.15.28:0/4289096093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4036070 con 0x7f60dc0cd490 2024-04-25T19:11:08.006 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.003+0000 7f60e1473640 1 -- 172.21.15.28:0/4289096093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d4031600 con 0x7f60dc0cd490 2024-04-25T19:11:08.006 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.003+0000 7f60e1473640 1 -- 172.21.15.28:0/4289096093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d40319d0 con 0x7f60dc0cd490 2024-04-25T19:11:08.006 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.003+0000 7f60e46d9640 1 -- 172.21.15.28:0/4289096093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0cd490 msgr2=0x7f60dc0cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:08.006 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.003+0000 7f60e46d9640 1 --2- 172.21.15.28:0/4289096093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0cd490 0x7f60dc0cd8a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0b8820 tx=0x7f60d402a610 comp rx=0 tx=0).stop 2024-04-25T19:11:08.006 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.004+0000 7f60e46d9640 1 -- 172.21.15.28:0/4289096093 shutdown_connections 2024-04-25T19:11:08.006 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.004+0000 7f60e46d9640 1 --2- 172.21.15.28:0/4289096093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0cd490 0x7f60dc0cd8a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.007 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.004+0000 7f60e46d9640 1 --2- 172.21.15.28:0/4289096093 >> v2:172.21.15.53:3300/0 conn(0x7f60dc0d20c0 0x7f60dc0ccf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.007 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.004+0000 7f60e46d9640 1 --2- 172.21.15.28:0/4289096093 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 0x7f60dc0d1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.007 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.004+0000 7f60e46d9640 1 -- 172.21.15.28:0/4289096093 >> 172.21.15.28:0/4289096093 conn(0x7f60dc09b540 msgr2=0x7f60dc09b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:08.007 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.004+0000 7f60e46d9640 1 -- 172.21.15.28:0/4289096093 shutdown_connections 2024-04-25T19:11:08.007 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.004+0000 7f60e46d9640 1 -- 172.21.15.28:0/4289096093 wait complete. 2024-04-25T19:11:08.007 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.005+0000 7f60e46d9640 1 Processor -- start 2024-04-25T19:11:08.008 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.005+0000 7f60e46d9640 1 -- start start 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e46d9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60dc0cd490 0x7f60dc0cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e46d9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 0x7f60dc0c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e46d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0d20c0 0x7f60dc1f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e46d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc0d24d0 con 0x7f60dc0d20c0 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e46d9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f60dc06c6a0 con 0x7f60dc0cd490 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e46d9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f60dc06ef90 con 0x7f60dc0cfca0 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0d20c0 0x7f60dc1f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2475640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60dc0cd490 0x7f60dc0cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0d20c0 0x7f60dc1f6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37958/0 (socket says 172.21.15.28:37958) 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 -- 172.21.15.28:0/3159871606 learned_addr learned my addr 172.21.15.28:0/3159871606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e1c74640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 0x7f60dc0c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:08.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 -- 172.21.15.28:0/3159871606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 msgr2=0x7f60dc0c9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 --2- 172.21.15.28:0/3159871606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 0x7f60dc0c9990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 -- 172.21.15.28:0/3159871606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60dc0cd490 msgr2=0x7f60dc0cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 --2- 172.21.15.28:0/3159871606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60dc0cd490 0x7f60dc0cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 -- 172.21.15.28:0/3159871606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60dc0c9ed0 con 0x7f60dc0d20c0 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60e2c76640 1 --2- 172.21.15.28:0/3159871606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0d20c0 0x7f60dc1f6000 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f60d4034040 tx=0x7f60d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.007+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3159871606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4036050 con 0x7f60dc0d20c0 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.008+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3159871606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60d4042070 con 0x7f60dc0d20c0 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.008+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60dc0cc4b0 con 0x7f60dc0d20c0 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.008+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3159871606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d4009510 con 0x7f60dc0d20c0 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.008+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60dc161ae0 con 0x7f60dc0d20c0 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.008+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3159871606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f60d4007bf0 con 0x7f60dc0d20c0 2024-04-25T19:11:08.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.008+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3159871606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f60d403db20 con 0x7f60dc0d20c0 2024-04-25T19:11:08.012 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.008+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60dc050740 con 0x7f60dc0d20c0 2024-04-25T19:11:08.012 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.011+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3159871606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f60d403ddc0 con 0x7f60dc0d20c0 2024-04-25T19:11:08.028 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:08.026+0000 7f69f4172040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:11:08.075 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.073+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f60dc06ea90 con 0x7f60dc0d20c0 2024-04-25T19:11:08.076 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.074+0000 7f60d2ffd640 1 -- 172.21.15.28:0/3159871606 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f60d4047460 con 0x7f60dc0d20c0 2024-04-25T19:11:08.076 INFO:teuthology.misc.health.smithi028.stdout: 2024-04-25T19:11:08.076 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":10,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:06.822340+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:08.077 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0d20c0 msgr2=0x7f60dc1f6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:08.077 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 --2- 172.21.15.28:0/3159871606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0d20c0 0x7f60dc1f6000 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f60d4034040 tx=0x7f60d4004a40 comp rx=0 tx=0).stop 2024-04-25T19:11:08.077 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 shutdown_connections 2024-04-25T19:11:08.077 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 --2- 172.21.15.28:0/3159871606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc0d20c0 0x7f60dc1f6000 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.078 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 --2- 172.21.15.28:0/3159871606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60dc0cfca0 0x7f60dc0c9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.078 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 --2- 172.21.15.28:0/3159871606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60dc0cd490 0x7f60dc0cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:08.078 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 >> 172.21.15.28:0/3159871606 conn(0x7f60dc09b540 msgr2=0x7f60dc0b15b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:08.078 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 shutdown_connections 2024-04-25T19:11:08.078 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:08.075+0000 7f60e46d9640 1 -- 172.21.15.28:0/3159871606 wait complete. 2024-04-25T19:11:08.080 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:08.079+0000 7f69f4172040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:11:08.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T19:11:08.095 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:08.093+0000 7f294431c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T19:11:08.147 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:08.145+0000 7f294431c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T19:11:08.154 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:08.153+0000 7f69f4172040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:11:08.202 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:08.200+0000 7f69f4172040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:11:08.220 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:08.219+0000 7f294431c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T19:11:08.268 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:08.266+0000 7f294431c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T19:11:08.705 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:08.703+0000 7f69f4172040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:11:08.771 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:08.769+0000 7f294431c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T19:11:08.825 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:08.824+0000 7f69f4172040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:11:08.896 INFO:tasks.ceph.mgr.x.smithi053.stderr:2024-04-25T19:11:08.894+0000 7f294431c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T19:11:09.841 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:09.839+0000 7f6994c00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T19:11:09.841 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-04-25T19:11:09.839+0000 7f6994c00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T19:11:10.967 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-04-25T19:11:10.962+0000 7fde0521a640 -1 osd.1 0 log_to_monitors true 2024-04-25T19:11:10.971 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-04-25T19:11:10.969+0000 7f21317d6640 -1 osd.0 0 log_to_monitors true 2024-04-25T19:11:10.993 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-04-25T19:11:10.990+0000 7fdff0216640 -1 osd.2 0 log_to_monitors true 2024-04-25T19:11:11.000 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-04-25T19:11:10.998+0000 7f4c21fcb640 -1 osd.6 0 log_to_monitors true 2024-04-25T19:11:11.003 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-04-25T19:11:11.001+0000 7ff1caa21640 -1 osd.7 0 log_to_monitors true 2024-04-25T19:11:11.007 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-04-25T19:11:11.005+0000 7f77edc2c640 -1 osd.5 0 log_to_monitors true 2024-04-25T19:11:11.013 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-04-25T19:11:11.012+0000 7f0c91611640 -1 osd.3 0 log_to_monitors true 2024-04-25T19:11:11.015 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-04-25T19:11:11.014+0000 7f2928816640 -1 osd.4 0 log_to_monitors true 2024-04-25T19:11:14.308 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-04-25T19:11:14.306+0000 7f4c1df61640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T19:11:14.331 INFO:tasks.ceph.osd.6.smithi053.stderr:2024-04-25T19:11:14.329+0000 7f4c15d5c640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:11:14.334 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-04-25T19:11:14.330+0000 7f212df80640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T19:11:14.351 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-04-25T19:11:14.350+0000 7ff1c69b7640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T19:11:14.352 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-04-25T19:11:14.350+0000 7f77e9bc2640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T19:11:14.354 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-04-25T19:11:14.352+0000 7f2125567640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:11:14.359 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-04-25T19:11:14.357+0000 7fde00be7640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T19:11:14.361 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-04-25T19:11:14.358+0000 7f0c8d7ac640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T19:11:14.365 INFO:tasks.ceph.osd.5.smithi053.stderr:2024-04-25T19:11:14.364+0000 7f77e19bd640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:11:14.366 INFO:tasks.ceph.osd.7.smithi053.stderr:2024-04-25T19:11:14.364+0000 7ff1be7b2640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:11:14.370 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-04-25T19:11:14.368+0000 7fddf89e2640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:11:14.381 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-04-25T19:11:14.378+0000 7f0c84d93640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:11:14.385 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-04-25T19:11:14.383+0000 7f2924180640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T19:11:14.398 INFO:tasks.ceph.osd.4.smithi053.stderr:2024-04-25T19:11:14.397+0000 7f291bf7b640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:11:14.431 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-04-25T19:11:14.428+0000 7fdfebaa5640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T19:11:14.442 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-04-25T19:11:14.440+0000 7fdfe38a0640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T19:11:14.893 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:15.007 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.006+0000 7fa88cf7d640 1 Processor -- start 2024-04-25T19:11:15.008 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.006+0000 7fa88cf7d640 1 -- start start 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.007+0000 7fa88cf7d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 0x7fa88809ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.007+0000 7fa88cf7d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8880b7840 0x7fa8880d2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.007+0000 7fa88cf7d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8880d1bc0 con 0x7fa8880d2170 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.007+0000 7fa88cf7d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fa8880cf6e0 con 0x7fa8880b7840 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.007+0000 7fa88cf7d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa8880b8f90 con 0x7fa8880d14f0 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.007+0000 7fa886d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8880b7840 0x7fa8880d2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.007+0000 7fa87ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa8880d2170 0x7fa8880b7130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37538/0 (socket says 172.21.15.28:37538) 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.007+0000 7fa87ffff640 1 -- 172.21.15.28:0/58696723 learned_addr learned my addr 172.21.15.28:0/58696723 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.008+0000 7fa88659c640 1 --2- 172.21.15.28:0/58696723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 0x7fa88809ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.008+0000 7fa886d9d640 1 -- 172.21.15.28:0/58696723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 msgr2=0x7fa88809ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:15.010 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.008+0000 7fa886d9d640 1 --2- 172.21.15.28:0/58696723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 0x7fa88809ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.008+0000 7fa886d9d640 1 -- 172.21.15.28:0/58696723 >> v1:172.21.15.28:6789/0 conn(0x7fa8880d2170 legacy=0x7fa8880b7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:15.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.008+0000 7fa885d9b640 1 -- 172.21.15.28:0/58696723 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3036109073 0 0) 0x7fa8880d1bc0 con 0x7fa8880d2170 2024-04-25T19:11:15.011 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.008+0000 7fa886d9d640 1 -- 172.21.15.28:0/58696723 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8880b91d0 con 0x7fa8880b7840 2024-04-25T19:11:15.012 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa886d9d640 1 --2- 172.21.15.28:0/58696723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8880d2580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa880009d90 tx=0x7fa88002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51cb4d16ad5edf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:15.012 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa885d9b640 1 -- 172.21.15.28:0/58696723 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880031600 con 0x7fa8880b7840 2024-04-25T19:11:15.012 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa885d9b640 1 -- 172.21.15.28:0/58696723 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8800317a0 con 0x7fa8880b7840 2024-04-25T19:11:15.012 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa885d9b640 1 -- 172.21.15.28:0/58696723 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880031b70 con 0x7fa8880b7840 2024-04-25T19:11:15.012 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa88cf7d640 1 -- 172.21.15.28:0/58696723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 msgr2=0x7fa8880d2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/58696723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8880d2580 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa880009d90 tx=0x7fa88002a5d0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa88cf7d640 1 -- 172.21.15.28:0/58696723 shutdown_connections 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/58696723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8880d2580 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/58696723 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 0x7fa88809ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.009+0000 7fa88cf7d640 1 -- 172.21.15.28:0/58696723 >> 172.21.15.28:0/58696723 conn(0x7fa8880d03a0 msgr2=0x7fa8880d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.010+0000 7fa88cf7d640 1 -- 172.21.15.28:0/58696723 shutdown_connections 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.010+0000 7fa88cf7d640 1 -- 172.21.15.28:0/58696723 wait complete. 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.012+0000 7fa88cf7d640 1 Processor -- start 2024-04-25T19:11:15.013 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.012+0000 7fa88cf7d640 1 -- start start 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88cf7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8881e85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88cf7d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 0x7fa8881d5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88cf7d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880d2170 0x7fa8880b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88cf7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8880d35b0 con 0x7fa8880b7840 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88cf7d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa8880b94b0 con 0x7fa8880d2170 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88cf7d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa8880d3730 con 0x7fa8880d14f0 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8881e85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8881e85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38060/0 (socket says 172.21.15.28:38060) 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88659c640 1 -- 172.21.15.28:0/3159675218 learned_addr learned my addr 172.21.15.28:0/3159675218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa886d9d640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880d2170 0x7fa8880b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:15.016 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa87ffff640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 0x7fa8881d5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88659c640 1 -- 172.21.15.28:0/3159675218 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 msgr2=0x7fa8881d5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88659c640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 0x7fa8881d5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88659c640 1 -- 172.21.15.28:0/3159675218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880d2170 msgr2=0x7fa8880b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88659c640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880d2170 0x7fa8880b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.013+0000 7fa88659c640 1 -- 172.21.15.28:0/3159675218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8881e7000 con 0x7fa8880b7840 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.014+0000 7fa88659c640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8881e85a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa87800e8a0 tx=0x7fa87800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.014+0000 7fa87effd640 1 -- 172.21.15.28:0/3159675218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878017020 con 0x7fa8880b7840 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.014+0000 7fa87effd640 1 -- 172.21.15.28:0/3159675218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87800ee90 con 0x7fa8880b7840 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.014+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8881d5fa0 con 0x7fa8880b7840 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.014+0000 7fa87effd640 1 -- 172.21.15.28:0/3159675218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878004460 con 0x7fa8880b7840 2024-04-25T19:11:15.017 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.014+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8881e77e0 con 0x7fa8880b7840 2024-04-25T19:11:15.018 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.015+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa888072bf0 con 0x7fa8880b7840 2024-04-25T19:11:15.018 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.016+0000 7fa87effd640 1 -- 172.21.15.28:0/3159675218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 96726+0+0 (secure 0 0 0) 0x7fa87801a020 con 0x7fa8880b7840 2024-04-25T19:11:15.018 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.016+0000 7fa87effd640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa868074420 0x7fa868076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:15.018 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.016+0000 7fa87effd640 1 -- 172.21.15.28:0/3159675218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7fa878094250 con 0x7fa8880b7840 2024-04-25T19:11:15.018 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.016+0000 7fa87ffff640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa868074420 0x7fa868076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:15.019 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.017+0000 7fa87ffff640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa868074420 0x7fa868076930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa8881e15a0 tx=0x7fa870007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:15.023 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.022+0000 7fa87effd640 1 -- 172.21.15.28:0/3159675218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa888072bf0 con 0x7fa8880b7840 2024-04-25T19:11:15.208 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.207+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa888050740 con 0x7fa8880b7840 2024-04-25T19:11:15.209 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.208+0000 7fa87effd640 1 -- 172.21.15.28:0/3159675218 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7fa878021030 con 0x7fa8880b7840 2024-04-25T19:11:15.209 INFO:teuthology.misc.health.smithi028.stdout: 2024-04-25T19:11:15.209 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":12,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:12.839140+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa868074420 msgr2=0x7fa868076930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa868074420 0x7fa868076930 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa8881e15a0 tx=0x7fa870007de0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 msgr2=0x7fa8881e85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8881e85a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa87800e8a0 tx=0x7fa87800ed60 comp rx=0 tx=0).stop 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 shutdown_connections 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa868074420 0x7fa868076930 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8880d2170 0x7fa8880b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8880d14f0 0x7fa8881d5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.212 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 --2- 172.21.15.28:0/3159675218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8880b7840 0x7fa8881e85a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:15.213 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 >> 172.21.15.28:0/3159675218 conn(0x7fa8880d03a0 msgr2=0x7fa8881e5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:15.213 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 shutdown_connections 2024-04-25T19:11:15.213 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:15.210+0000 7fa88cf7d640 1 -- 172.21.15.28:0/3159675218 wait complete. 2024-04-25T19:11:15.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T19:11:22.029 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:22.146 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.144+0000 7f8b13692640 1 Processor -- start 2024-04-25T19:11:22.146 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.144+0000 7f8b13692640 1 -- start start 2024-04-25T19:11:22.148 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b13692640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8b0c0b4e70 0x7f8b0c164970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.148 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b13692640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0d0960 0x7f8b0c0b7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.148 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b13692640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b0c0d1300 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b13692640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8b0c163e90 con 0x7f8b0c0b4e70 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b13692640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8b0c164170 con 0x7f8b0c0d0960 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b1142e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8b0c0b4e70 0x7f8b0c164970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b1142e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8b0c0b4e70 0x7f8b0c164970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55922/0 (socket says 172.21.15.28:55922) 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b1142e640 1 -- 172.21.15.28:0/2772375198 learned_addr learned my addr 172.21.15.28:0/2772375198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b10c2d640 1 --2- 172.21.15.28:0/2772375198 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0d0960 0x7f8b0c0b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b1142e640 1 -- 172.21.15.28:0/2772375198 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0d0960 msgr2=0x7f8b0c0b7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b1142e640 1 --2- 172.21.15.28:0/2772375198 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0d0960 0x7f8b0c0b7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b1142e640 1 -- 172.21.15.28:0/2772375198 >> v1:172.21.15.53:6789/0 conn(0x7f8b0c0d0d70 legacy=0x7f8b0c06dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.146+0000 7f8b1142e640 1 -- 172.21.15.28:0/2772375198 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0c164450 con 0x7f8b0c0b4e70 2024-04-25T19:11:22.149 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.147+0000 7f8b1142e640 1 --2- 172.21.15.28:0/2772375198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0b4e70 0x7f8b0c164970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009870 tx=0x7f8b0402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa7d78c97d16c2d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.147+0000 7f8b0bfff640 1 -- 172.21.15.28:0/2772375198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04037070 con 0x7f8b0c0b4e70 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.147+0000 7f8b0bfff640 1 -- 172.21.15.28:0/2772375198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b04032440 con 0x7f8b0c0b4e70 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.147+0000 7f8b0bfff640 1 -- 172.21.15.28:0/2772375198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b04032810 con 0x7f8b0c0b4e70 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.148+0000 7f8b13692640 1 -- 172.21.15.28:0/2772375198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0b4e70 msgr2=0x7f8b0c164970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.148+0000 7f8b13692640 1 --2- 172.21.15.28:0/2772375198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0b4e70 0x7f8b0c164970 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009870 tx=0x7f8b0402a1c0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.148+0000 7f8b13692640 1 -- 172.21.15.28:0/2772375198 shutdown_connections 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.148+0000 7f8b13692640 1 --2- 172.21.15.28:0/2772375198 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0d0960 0x7f8b0c0b7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.148+0000 7f8b13692640 1 --2- 172.21.15.28:0/2772375198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0b4e70 0x7f8b0c164970 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.148+0000 7f8b13692640 1 -- 172.21.15.28:0/2772375198 >> 172.21.15.28:0/2772375198 conn(0x7f8b0c0d2630 msgr2=0x7f8b0c0d2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:22.151 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.148+0000 7f8b13692640 1 -- 172.21.15.28:0/2772375198 shutdown_connections 2024-04-25T19:11:22.152 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.148+0000 7f8b13692640 1 -- 172.21.15.28:0/2772375198 wait complete. 2024-04-25T19:11:22.154 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.150+0000 7f8b13692640 1 Processor -- start 2024-04-25T19:11:22.154 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.150+0000 7f8b13692640 1 -- start start 2024-04-25T19:11:22.154 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b13692640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0b4e70 0x7f8b0c148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.154 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b13692640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b0c0d0960 0x7f8b0c162d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.154 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b13692640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0d0d70 0x7f8b0c145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.154 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b13692640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0c0d54b0 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.154 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b13692640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8b0c0d5330 con 0x7f8b0c0d0960 2024-04-25T19:11:22.154 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b13692640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8b0c0b7b70 con 0x7f8b0c0b4e70 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b11c2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0d0d70 0x7f8b0c145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b11c2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0d0d70 0x7f8b0c145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55928/0 (socket says 172.21.15.28:55928) 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b11c2f640 1 -- 172.21.15.28:0/2358754709 learned_addr learned my addr 172.21.15.28:0/2358754709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b10c2d640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b0c0d0960 0x7f8b0c162d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b1142e640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0b4e70 0x7f8b0c148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b11c2f640 1 -- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0b4e70 msgr2=0x7f8b0c148dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b11c2f640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0b4e70 0x7f8b0c148dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b11c2f640 1 -- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b0c0d0960 msgr2=0x7f8b0c162d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b11c2f640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b0c0d0960 0x7f8b0c162d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b11c2f640 1 -- 172.21.15.28:0/2358754709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0c149300 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.155 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.151+0000 7f8b10c2d640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b0c0d0960 0x7f8b0c162d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:22.156 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.152+0000 7f8b11c2f640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0d0d70 0x7f8b0c145660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c6b0 tx=0x7f8afc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.156 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.152+0000 7f8b09ffb640 1 -- 172.21.15.28:0/2358754709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc003b70 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.156 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.152+0000 7f8b09ffb640 1 -- 172.21.15.28:0/2358754709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8afc003d10 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.156 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.152+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b0c1f9a80 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.156 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.152+0000 7f8b09ffb640 1 -- 172.21.15.28:0/2358754709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc005310 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.156 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.152+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b0c1f9d10 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.157 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.153+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b0c050740 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.157 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.154+0000 7f8b09ffb640 1 -- 172.21.15.28:0/2358754709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8afc0054b0 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.157 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.154+0000 7f8b09ffb640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8af80744e0 0x7f8af80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.157 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.154+0000 7f8b09ffb640 1 -- 172.21.15.28:0/2358754709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f8afc094be0 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.157 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.154+0000 7f8b1142e640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8af80744e0 0x7f8af80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.157 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.155+0000 7f8b1142e640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8af80744e0 0x7f8af80769f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8b04034000 tx=0x7f8b0402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.161 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.159+0000 7f8b09ffb640 1 -- 172.21.15.28:0/2358754709 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b0c149300 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.346 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.344+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b0c156670 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.347 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.345+0000 7f8b09ffb640 1 -- 172.21.15.28:0/2358754709 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12824 (secure 0 0 0) 0x7f8afc05f750 con 0x7f8b0c0d0d70 2024-04-25T19:11:22.347 INFO:teuthology.misc.health.smithi028.stdout: 2024-04-25T19:11:22.347 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":16,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:18.881227+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:22.349 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8af80744e0 msgr2=0x7f8af80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.349 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8af80744e0 0x7f8af80769f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8b04034000 tx=0x7f8b0402e000 comp rx=0 tx=0).stop 2024-04-25T19:11:22.349 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0d0d70 msgr2=0x7f8b0c145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.349 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0d0d70 0x7f8b0c145660 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00c6b0 tx=0x7f8afc00cb70 comp rx=0 tx=0).stop 2024-04-25T19:11:22.350 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 shutdown_connections 2024-04-25T19:11:22.350 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8af80744e0 0x7f8af80769f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.350 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c0d0d70 0x7f8b0c145660 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.350 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b0c0d0960 0x7f8b0c162d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.350 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 --2- 172.21.15.28:0/2358754709 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8b0c0b4e70 0x7f8b0c148dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.350 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 >> 172.21.15.28:0/2358754709 conn(0x7f8b0c0d2630 msgr2=0x7f8b0c0b6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:22.350 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 shutdown_connections 2024-04-25T19:11:22.350 INFO:teuthology.misc.health.smithi028.stderr:2024-04-25T19:11:22.347+0000 7f8b13692640 1 -- 172.21.15.28:0/2358754709 wait complete. 2024-04-25T19:11:22.359 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T19:11:22.359 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T19:11:22.359 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T19:11:22.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T19:11:22.362 INFO:tasks.ceph.mds.a:Started 2024-04-25T19:11:22.362 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T19:11:22.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T19:11:22.364 INFO:tasks.ceph.mds.c:Started 2024-04-25T19:11:22.364 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T19:11:22.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T19:11:22.367 INFO:tasks.ceph.mds.b:Started 2024-04-25T19:11:22.367 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T19:11:22.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T19:11:22.370 INFO:tasks.ceph.mds.d:Started 2024-04-25T19:11:22.370 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T19:11:22.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:22.450 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-04-25T19:11:22.451 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-04-25T19:11:22.469 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-04-25T19:11:22.470 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-04-25T19:11:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.484+0000 7f5d437ce640 1 Processor -- start 2024-04-25T19:11:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.484+0000 7f5d437ce640 1 -- start start 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d437ce640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d3c06e270 0x7f5d3c06e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d437ce640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 0x7f5d3c164d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d437ce640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d3c0d32c0 con 0x7f5d3c165260 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d437ce640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3c0b9200 con 0x7f5d3c06e270 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d437ce640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5d3c0d10a0 con 0x7f5d3c164910 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d4156a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d3c06e270 0x7f5d3c06e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d41d6b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5d3c165260 0x7f5d3c0ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53588/0 (socket says 172.21.15.28:53588) 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d4156a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d3c06e270 0x7f5d3c06e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55980/0 (socket says 172.21.15.28:55980) 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d41d6b640 1 -- 172.21.15.28:0/1410598198 learned_addr learned my addr 172.21.15.28:0/1410598198 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:22.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d40d69640 1 --2- 172.21.15.28:0/1410598198 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 0x7f5d3c164d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d4156a640 1 -- 172.21.15.28:0/1410598198 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 msgr2=0x7f5d3c164d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d4156a640 1 --2- 172.21.15.28:0/1410598198 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 0x7f5d3c164d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d4156a640 1 -- 172.21.15.28:0/1410598198 >> v1:172.21.15.28:6789/0 conn(0x7f5d3c165260 legacy=0x7f5d3c0ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d33fff640 1 -- 172.21.15.28:0/1410598198 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 94420913 0 0) 0x7f5d34001490 con 0x7f5d3c165260 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.486+0000 7f5d4156a640 1 -- 172.21.15.28:0/1410598198 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3c165670 con 0x7f5d3c06e270 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.487+0000 7f5d4156a640 1 --2- 172.21.15.28:0/1410598198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c06e270 0x7f5d3c06e680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0097a0 tx=0x7f5d2c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999f851109369309 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.487+0000 7f5d33fff640 1 -- 172.21.15.28:0/1410598198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c037070 con 0x7f5d3c06e270 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.487+0000 7f5d33fff640 1 -- 172.21.15.28:0/1410598198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d2c032440 con 0x7f5d3c06e270 2024-04-25T19:11:22.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.487+0000 7f5d33fff640 1 -- 172.21.15.28:0/1410598198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c032810 con 0x7f5d3c06e270 2024-04-25T19:11:22.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.488+0000 7f5d437ce640 1 -- 172.21.15.28:0/1410598198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c06e270 msgr2=0x7f5d3c06e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.488+0000 7f5d437ce640 1 --2- 172.21.15.28:0/1410598198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c06e270 0x7f5d3c06e680 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c0097a0 tx=0x7f5d2c02a180 comp rx=0 tx=0).stop 2024-04-25T19:11:22.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.488+0000 7f5d437ce640 1 -- 172.21.15.28:0/1410598198 shutdown_connections 2024-04-25T19:11:22.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.488+0000 7f5d437ce640 1 --2- 172.21.15.28:0/1410598198 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 0x7f5d3c164d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.488+0000 7f5d437ce640 1 --2- 172.21.15.28:0/1410598198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c06e270 0x7f5d3c06e680 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.488+0000 7f5d437ce640 1 -- 172.21.15.28:0/1410598198 >> 172.21.15.28:0/1410598198 conn(0x7f5d3c0d03e0 msgr2=0x7f5d3c0d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:22.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.488+0000 7f5d437ce640 1 -- 172.21.15.28:0/1410598198 shutdown_connections 2024-04-25T19:11:22.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.488+0000 7f5d437ce640 1 -- 172.21.15.28:0/1410598198 wait complete. 2024-04-25T19:11:22.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.490+0000 7f5d437ce640 1 Processor -- start 2024-04-25T19:11:22.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.490+0000 7f5d437ce640 1 -- start start 2024-04-25T19:11:22.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d437ce640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d3c06e270 0x7f5d3c14f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d437ce640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 0x7f5d3c15a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d437ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c165260 0x7f5d3c15a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d437ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c09d2d0 con 0x7f5d3c165260 2024-04-25T19:11:22.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d437ce640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c09d150 con 0x7f5d3c06e270 2024-04-25T19:11:22.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d437ce640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5d3c06db40 con 0x7f5d3c164910 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d41d6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c165260 0x7f5d3c15a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d41d6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c165260 0x7f5d3c15a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55984/0 (socket says 172.21.15.28:55984) 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d41d6b640 1 -- 172.21.15.28:0/759975871 learned_addr learned my addr 172.21.15.28:0/759975871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d4156a640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d3c06e270 0x7f5d3c14f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d40d69640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 0x7f5d3c15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d41d6b640 1 -- 172.21.15.28:0/759975871 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 msgr2=0x7f5d3c15a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d41d6b640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 0x7f5d3c15a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d41d6b640 1 -- 172.21.15.28:0/759975871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d3c06e270 msgr2=0x7f5d3c14f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d41d6b640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d3c06e270 0x7f5d3c14f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.491+0000 7f5d41d6b640 1 -- 172.21.15.28:0/759975871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3c15afb0 con 0x7f5d3c165260 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.492+0000 7f5d41d6b640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c165260 0x7f5d3c15a770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400c700 tx=0x7f5d3400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.492+0000 7f5d31ffb640 1 -- 172.21.15.28:0/759975871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d34018070 con 0x7f5d3c165260 2024-04-25T19:11:22.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.492+0000 7f5d31ffb640 1 -- 172.21.15.28:0/759975871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3400ce10 con 0x7f5d3c165260 2024-04-25T19:11:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.492+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d3c15ce40 con 0x7f5d3c165260 2024-04-25T19:11:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.492+0000 7f5d31ffb640 1 -- 172.21.15.28:0/759975871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3400a880 con 0x7f5d3c165260 2024-04-25T19:11:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.492+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d3c15e9a0 con 0x7f5d3c165260 2024-04-25T19:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.493+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d3c050740 con 0x7f5d3c165260 2024-04-25T19:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.494+0000 7f5d31ffb640 1 -- 172.21.15.28:0/759975871 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5d3401b070 con 0x7f5d3c165260 2024-04-25T19:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.494+0000 7f5d31ffb640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d200744b0 0x7f5d200769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.494+0000 7f5d31ffb640 1 -- 172.21.15.28:0/759975871 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f5d340943e0 con 0x7f5d3c165260 2024-04-25T19:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.494+0000 7f5d4156a640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d200744b0 0x7f5d200769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.494+0000 7f5d4156a640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d200744b0 0x7f5d200769c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c034000 tx=0x7f5d2c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.499+0000 7f5d31ffb640 1 -- 172.21.15.28:0/759975871 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d3c0d32c0 con 0x7f5d3c165260 2024-04-25T19:11:22.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.688+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d3c0d1ab0 con 0x7f5d3c165260 2024-04-25T19:11:22.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.689+0000 7f5d31ffb640 1 -- 172.21.15.28:0/759975871 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12824 (secure 0 0 0) 0x7f5d3405e860 con 0x7f5d3c165260 2024-04-25T19:11:22.691 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:22.691 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:18.881227+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d200744b0 msgr2=0x7f5d200769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d200744b0 0x7f5d200769c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c034000 tx=0x7f5d2c02e000 comp rx=0 tx=0).stop 2024-04-25T19:11:22.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c165260 msgr2=0x7f5d3c15a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c165260 0x7f5d3c15a770 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400c700 tx=0x7f5d3400cbc0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 shutdown_connections 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d200744b0 0x7f5d200769c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d3c165260 0x7f5d3c15a770 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d3c164910 0x7f5d3c15a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 --2- 172.21.15.28:0/759975871 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d3c06e270 0x7f5d3c14f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 >> 172.21.15.28:0/759975871 conn(0x7f5d3c0d03e0 msgr2=0x7f5d3c0715d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 shutdown_connections 2024-04-25T19:11:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.691+0000 7f5d437ce640 1 -- 172.21.15.28:0/759975871 wait complete. 2024-04-25T19:11:22.704 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:11:22.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:11:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.817+0000 7fb27e49f640 1 Processor -- start 2024-04-25T19:11:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.817+0000 7fb27e49f640 1 -- start start 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb27e49f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 0x7fb27815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb27e49f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2780b7660 0x7fb2780b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb27e49f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb278161210 con 0x7fb2781606f0 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb27e49f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb2780b7fb0 con 0x7fb2780b7660 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb27e49f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb2781614a0 con 0x7fb278164990 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb2777fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2780b7660 0x7fb2780b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb27ca3c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb2781606f0 0x7fb278160b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53590/0 (socket says 172.21.15.28:53590) 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb2777fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb2780b7660 0x7fb2780b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55988/0 (socket says 172.21.15.28:55988) 2024-04-25T19:11:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb27ca3c640 1 -- 172.21.15.28:0/3635678917 learned_addr learned my addr 172.21.15.28:0/3635678917 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb277fff640 1 --2- 172.21.15.28:0/3635678917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 0x7fb27815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb2777fe640 1 -- 172.21.15.28:0/3635678917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 msgr2=0x7fb27815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb2777fe640 1 --2- 172.21.15.28:0/3635678917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 0x7fb27815dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb2777fe640 1 -- 172.21.15.28:0/3635678917 >> v1:172.21.15.28:6789/0 conn(0x7fb2781606f0 legacy=0x7fb278160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb2777fe640 1 -- 172.21.15.28:0/3635678917 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb278163eb0 con 0x7fb2780b7660 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.818+0000 7fb276ffd640 1 -- 172.21.15.28:0/3635678917 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3543107099 0 0) 0x7fb278161210 con 0x7fb2781606f0 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.819+0000 7fb2777fe640 1 --2- 172.21.15.28:0/3635678917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2780b7660 0x7fb2780b7a70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb2680097a0 tx=0x7fb26802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da99bf43dd646ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.819+0000 7fb276ffd640 1 -- 172.21.15.28:0/3635678917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268037070 con 0x7fb2780b7660 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.819+0000 7fb276ffd640 1 -- 172.21.15.28:0/3635678917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb268032440 con 0x7fb2780b7660 2024-04-25T19:11:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.819+0000 7fb276ffd640 1 -- 172.21.15.28:0/3635678917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268032810 con 0x7fb2780b7660 2024-04-25T19:11:22.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.820+0000 7fb27e49f640 1 -- 172.21.15.28:0/3635678917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2780b7660 msgr2=0x7fb2780b7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.820+0000 7fb27e49f640 1 --2- 172.21.15.28:0/3635678917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2780b7660 0x7fb2780b7a70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb2680097a0 tx=0x7fb26802a180 comp rx=0 tx=0).stop 2024-04-25T19:11:22.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.820+0000 7fb27e49f640 1 -- 172.21.15.28:0/3635678917 shutdown_connections 2024-04-25T19:11:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.820+0000 7fb27e49f640 1 --2- 172.21.15.28:0/3635678917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2780b7660 0x7fb2780b7a70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.820+0000 7fb27e49f640 1 --2- 172.21.15.28:0/3635678917 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 0x7fb27815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.820+0000 7fb27e49f640 1 -- 172.21.15.28:0/3635678917 >> 172.21.15.28:0/3635678917 conn(0x7fb2780d2840 msgr2=0x7fb2780d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.820+0000 7fb27e49f640 1 -- 172.21.15.28:0/3635678917 shutdown_connections 2024-04-25T19:11:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.820+0000 7fb27e49f640 1 -- 172.21.15.28:0/3635678917 wait complete. 2024-04-25T19:11:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.822+0000 7fb27e49f640 1 Processor -- start 2024-04-25T19:11:22.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.822+0000 7fb27e49f640 1 -- start start 2024-04-25T19:11:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb27e49f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2780b7660 0x7fb2781eae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb27e49f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2781606f0 0x7fb2780c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb27e49f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 0x7fb27815e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb27e49f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2780b45d0 con 0x7fb2781606f0 2024-04-25T19:11:22.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb27e49f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb278164470 con 0x7fb2780b7660 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb27e49f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb2780b4750 con 0x7fb278164990 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb2777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2781606f0 0x7fb2780c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb2777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2781606f0 0x7fb2780c8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55992/0 (socket says 172.21.15.28:55992) 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.823+0000 7fb2777fe640 1 -- 172.21.15.28:0/832771210 learned_addr learned my addr 172.21.15.28:0/832771210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb277fff640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2780b7660 0x7fb2781eae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb27ca3c640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 0x7fb27815e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb2777fe640 1 -- 172.21.15.28:0/832771210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 msgr2=0x7fb27815e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb2777fe640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 0x7fb27815e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb2777fe640 1 -- 172.21.15.28:0/832771210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2780b7660 msgr2=0x7fb2781eae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb2777fe640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2780b7660 0x7fb2781eae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb2777fe640 1 -- 172.21.15.28:0/832771210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2781eb380 con 0x7fb2781606f0 2024-04-25T19:11:22.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb2777fe640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2781606f0 0x7fb2780c8c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb268000c00 tx=0x7fb2680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb274ff9640 1 -- 172.21.15.28:0/832771210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268037040 con 0x7fb2781606f0 2024-04-25T19:11:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb274ff9640 1 -- 172.21.15.28:0/832771210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb268042070 con 0x7fb2781606f0 2024-04-25T19:11:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2780c9960 con 0x7fb2781606f0 2024-04-25T19:11:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb274ff9640 1 -- 172.21.15.28:0/832771210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268030a70 con 0x7fb2781606f0 2024-04-25T19:11:22.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.824+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2780c9f20 con 0x7fb2781606f0 2024-04-25T19:11:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.825+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb278050740 con 0x7fb2781606f0 2024-04-25T19:11:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.826+0000 7fb274ff9640 1 -- 172.21.15.28:0/832771210 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb268030c10 con 0x7fb2781606f0 2024-04-25T19:11:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.826+0000 7fb274ff9640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2640744b0 0x7fb2640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.826+0000 7fb274ff9640 1 -- 172.21.15.28:0/832771210 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fb2680b5a60 con 0x7fb2781606f0 2024-04-25T19:11:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.826+0000 7fb277fff640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2640744b0 0x7fb2640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:22.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.827+0000 7fb277fff640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2640744b0 0x7fb2640769c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb26c006fd0 tx=0x7fb26c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:22.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:22.831+0000 7fb274ff9640 1 -- 172.21.15.28:0/832771210 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb2781eb380 con 0x7fb2781606f0 2024-04-25T19:11:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.015+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb2780d4000 con 0x7fb2781606f0 2024-04-25T19:11:23.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.016+0000 7fb274ff9640 1 -- 172.21.15.28:0/832771210 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fb2680804c0 con 0x7fb2781606f0 2024-04-25T19:11:23.018 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:11:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2640744b0 msgr2=0x7fb2640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2640744b0 0x7fb2640769c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb26c006fd0 tx=0x7fb26c008040 comp rx=0 tx=0).stop 2024-04-25T19:11:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2781606f0 msgr2=0x7fb2780c8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2781606f0 0x7fb2780c8c40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb268000c00 tx=0x7fb2680046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 shutdown_connections 2024-04-25T19:11:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2640744b0 0x7fb2640769c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb278164990 0x7fb27815e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2781606f0 0x7fb2780c8c40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 --2- 172.21.15.28:0/832771210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2780b7660 0x7fb2781eae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 >> 172.21.15.28:0/832771210 conn(0x7fb2780d2840 msgr2=0x7fb2780b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 shutdown_connections 2024-04-25T19:11:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.018+0000 7fb27e49f640 1 -- 172.21.15.28:0/832771210 wait complete. 2024-04-25T19:11:23.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:11:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.140+0000 7f5e73140640 1 Processor -- start 2024-04-25T19:11:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.140+0000 7f5e73140640 1 -- start start 2024-04-25T19:11:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.141+0000 7f5e73140640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5e6c0b4e30 0x7f5e6c0bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.141+0000 7f5e73140640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b9470 0x7f5e6c0b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.141+0000 7f5e73140640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e6c0b7650 0x7f5e6c0b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.141+0000 7f5e73140640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f5e6c0d0b50 con 0x7f5e6c0b7650 2024-04-25T19:11:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.141+0000 7f5e73140640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5e6c08e3a0 con 0x7f5e6c0b4e30 2024-04-25T19:11:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.141+0000 7f5e73140640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5e6c0bbb90 con 0x7f5e6c0b9470 2024-04-25T19:11:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e716dd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e6c0b7650 0x7f5e6c0b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e716dd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e6c0b7650 0x7f5e6c0b7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56006/0 (socket says 172.21.15.28:56006) 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e716dd640 1 -- 172.21.15.28:0/3887204702 learned_addr learned my addr 172.21.15.28:0/3887204702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e70edc640 1 --2- 172.21.15.28:0/3887204702 >> v2:172.21.15.53:3300/0 conn(0x7f5e6c0b4e30 0x7f5e6c0bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e63fff640 1 --2- 172.21.15.28:0/3887204702 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b9470 0x7f5e6c0b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e716dd640 1 -- 172.21.15.28:0/3887204702 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b9470 msgr2=0x7f5e6c0b9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e716dd640 1 --2- 172.21.15.28:0/3887204702 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b9470 0x7f5e6c0b9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e716dd640 1 -- 172.21.15.28:0/3887204702 >> v2:172.21.15.53:3300/0 conn(0x7f5e6c0b4e30 msgr2=0x7f5e6c0bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e716dd640 1 --2- 172.21.15.28:0/3887204702 >> v2:172.21.15.53:3300/0 conn(0x7f5e6c0b4e30 0x7f5e6c0bb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.142+0000 7f5e716dd640 1 -- 172.21.15.28:0/3887204702 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6c0b7fa0 con 0x7f5e6c0b7650 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.143+0000 7f5e716dd640 1 --2- 172.21.15.28:0/3887204702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c0b7a60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c08e8e0 tx=0x7f5e6402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05eccf19e4a281f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.143+0000 7f5e637fe640 1 -- 172.21.15.28:0/3887204702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e64036070 con 0x7f5e6c0b7650 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.143+0000 7f5e637fe640 1 -- 172.21.15.28:0/3887204702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e64031600 con 0x7f5e6c0b7650 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.143+0000 7f5e637fe640 1 -- 172.21.15.28:0/3887204702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e640319d0 con 0x7f5e6c0b7650 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 -- 172.21.15.28:0/3887204702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 msgr2=0x7f5e6c0b7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 --2- 172.21.15.28:0/3887204702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c0b7a60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c08e8e0 tx=0x7f5e6402a610 comp rx=0 tx=0).stop 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 -- 172.21.15.28:0/3887204702 shutdown_connections 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 --2- 172.21.15.28:0/3887204702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c0b7a60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 --2- 172.21.15.28:0/3887204702 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b9470 0x7f5e6c0b9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 --2- 172.21.15.28:0/3887204702 >> v2:172.21.15.53:3300/0 conn(0x7f5e6c0b4e30 0x7f5e6c0bb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 -- 172.21.15.28:0/3887204702 >> 172.21.15.28:0/3887204702 conn(0x7f5e6c0d2650 msgr2=0x7f5e6c0d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 -- 172.21.15.28:0/3887204702 shutdown_connections 2024-04-25T19:11:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.144+0000 7f5e73140640 1 -- 172.21.15.28:0/3887204702 wait complete. 2024-04-25T19:11:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.145+0000 7f5e73140640 1 Processor -- start 2024-04-25T19:11:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.145+0000 7f5e73140640 1 -- start start 2024-04-25T19:11:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e73140640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b4e30 0x7f5e6c158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e73140640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c163130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e73140640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6c0b9470 0x7f5e6c154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e73140640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c0d0500 con 0x7f5e6c0b7650 2024-04-25T19:11:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e73140640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5e6c0d0220 con 0x7f5e6c0b9470 2024-04-25T19:11:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e73140640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5e6c0d07e0 con 0x7f5e6c0b4e30 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c163130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c163130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56012/0 (socket says 172.21.15.28:56012) 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 -- 172.21.15.28:0/748668518 learned_addr learned my addr 172.21.15.28:0/748668518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e70edc640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b4e30 0x7f5e6c158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e716dd640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6c0b9470 0x7f5e6c154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 -- 172.21.15.28:0/748668518 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b4e30 msgr2=0x7f5e6c158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b4e30 0x7f5e6c158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 -- 172.21.15.28:0/748668518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6c0b9470 msgr2=0x7f5e6c154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6c0b9470 0x7f5e6c154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 -- 172.21.15.28:0/748668518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e6c158da0 con 0x7f5e6c0b7650 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.147+0000 7f5e63fff640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c163130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00e870 tx=0x7f5e5c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.148+0000 7f5e617fa640 1 -- 172.21.15.28:0/748668518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c017070 con 0x7f5e6c0b7650 2024-04-25T19:11:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.148+0000 7f5e617fa640 1 -- 172.21.15.28:0/748668518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e5c00ee60 con 0x7f5e6c0b7650 2024-04-25T19:11:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.148+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e6c1ea000 con 0x7f5e6c0b7650 2024-04-25T19:11:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.148+0000 7f5e617fa640 1 -- 172.21.15.28:0/748668518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c00cb40 con 0x7f5e6c0b7650 2024-04-25T19:11:23.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.148+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e6c1ea5c0 con 0x7f5e6c0b7650 2024-04-25T19:11:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.149+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e6c1e9d60 con 0x7f5e6c0b7650 2024-04-25T19:11:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.149+0000 7f5e617fa640 1 -- 172.21.15.28:0/748668518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5e5c01a070 con 0x7f5e6c0b7650 2024-04-25T19:11:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.150+0000 7f5e617fa640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5e580744b0 0x7f5e580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.150+0000 7f5e70edc640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5e580744b0 0x7f5e580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.150+0000 7f5e617fa640 1 -- 172.21.15.28:0/748668518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f5e5c094f20 con 0x7f5e6c0b7650 2024-04-25T19:11:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.150+0000 7f5e70edc640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5e580744b0 0x7f5e580769c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0d2620 tx=0x7f5e68009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.155+0000 7f5e617fa640 1 -- 172.21.15.28:0/748668518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e6c1e9d60 con 0x7f5e6c0b7650 2024-04-25T19:11:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.375+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5e6c050740 con 0x7f5e6c0b7650 2024-04-25T19:11:23.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.375+0000 7f5e617fa640 1 -- 172.21.15.28:0/748668518 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7f5e5c05f440 con 0x7f5e6c0b7650 2024-04-25T19:11:23.377 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:23.378 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T19:11:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-04-25T19:11:23.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5e580744b0 msgr2=0x7f5e580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:23.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5e580744b0 0x7f5e580769c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c0d2620 tx=0x7f5e68009190 comp rx=0 tx=0).stop 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 msgr2=0x7f5e6c163130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c163130 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00e870 tx=0x7f5e5c00ed30 comp rx=0 tx=0).stop 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 shutdown_connections 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5e580744b0 0x7f5e580769c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5e6c0b9470 0x7f5e6c154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b7650 0x7f5e6c163130 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 --2- 172.21.15.28:0/748668518 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5e6c0b4e30 0x7f5e6c158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 >> 172.21.15.28:0/748668518 conn(0x7f5e6c0d2650 msgr2=0x7f5e6c0b5310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.377+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 shutdown_connections 2024-04-25T19:11:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:23.378+0000 7f5e73140640 1 -- 172.21.15.28:0/748668518 wait complete. 2024-04-25T19:11:25.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:11:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.515+0000 7f736cf1f640 1 Processor -- start 2024-04-25T19:11:25.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.515+0000 7f736cf1f640 1 -- start start 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f736cf1f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f73680b7d70 0x7f736815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f736cf1f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73681606f0 0x7f7368160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f736cf1f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73681614a0 con 0x7f7368164990 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f736cf1f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f7368163eb0 con 0x7f73680b7d70 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f736cf1f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7368164190 con 0x7f73681606f0 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f735ffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f73680b7d70 0x7f736815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f736659c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7368164990 0x7f73680b7660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53596/0 (socket says 172.21.15.28:53596) 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f735ffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f73680b7d70 0x7f736815dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56016/0 (socket says 172.21.15.28:56016) 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.516+0000 7f736659c640 1 -- 172.21.15.28:0/182487230 learned_addr learned my addr 172.21.15.28:0/182487230 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f7366d9d640 1 --2- 172.21.15.28:0/182487230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73681606f0 0x7f7368160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f735ffff640 1 -- 172.21.15.28:0/182487230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73681606f0 msgr2=0x7f7368160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f735ffff640 1 --2- 172.21.15.28:0/182487230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73681606f0 0x7f7368160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f735ffff640 1 -- 172.21.15.28:0/182487230 >> v1:172.21.15.28:6789/0 conn(0x7f7368164990 legacy=0x7f73680b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f735ffff640 1 -- 172.21.15.28:0/182487230 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7368164470 con 0x7f73680b7d70 2024-04-25T19:11:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f7365d9b640 1 -- 172.21.15.28:0/182487230 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572329969 0 0) 0x7f73681614a0 con 0x7f7368164990 2024-04-25T19:11:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f7366d9d640 1 --2- 172.21.15.28:0/182487230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73681606f0 0x7f7368160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:25.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f735ffff640 1 --2- 172.21.15.28:0/182487230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73680b7d70 0x7f736815dcb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73500097a0 tx=0x7f735002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de33fe7356401c4d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.517+0000 7f7365d9b640 1 -- 172.21.15.28:0/182487230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350037070 con 0x7f73680b7d70 2024-04-25T19:11:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f7365d9b640 1 -- 172.21.15.28:0/182487230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7350032440 con 0x7f73680b7d70 2024-04-25T19:11:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f7365d9b640 1 -- 172.21.15.28:0/182487230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350032810 con 0x7f73680b7d70 2024-04-25T19:11:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f736cf1f640 1 -- 172.21.15.28:0/182487230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73680b7d70 msgr2=0x7f736815dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f736cf1f640 1 --2- 172.21.15.28:0/182487230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73680b7d70 0x7f736815dcb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73500097a0 tx=0x7f735002a180 comp rx=0 tx=0).stop 2024-04-25T19:11:25.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f736cf1f640 1 -- 172.21.15.28:0/182487230 shutdown_connections 2024-04-25T19:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f736cf1f640 1 --2- 172.21.15.28:0/182487230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73681606f0 0x7f7368160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f736cf1f640 1 --2- 172.21.15.28:0/182487230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73680b7d70 0x7f736815dcb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f736cf1f640 1 -- 172.21.15.28:0/182487230 >> 172.21.15.28:0/182487230 conn(0x7f73680d2840 msgr2=0x7f73680d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f736cf1f640 1 -- 172.21.15.28:0/182487230 shutdown_connections 2024-04-25T19:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.518+0000 7f736cf1f640 1 -- 172.21.15.28:0/182487230 wait complete. 2024-04-25T19:11:25.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.520+0000 7f736cf1f640 1 Processor -- start 2024-04-25T19:11:25.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.520+0000 7f736cf1f640 1 -- start start 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.521+0000 7f736cf1f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73680b7d70 0x7f7368152f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.521+0000 7f736cf1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681606f0 0x7f73680bacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f736cf1f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7368164990 0x7f73680cf350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f736cf1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f73680b4750 con 0x7f73681606f0 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f736cf1f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f73680b45d0 con 0x7f7368164990 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f736cf1f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f73680b48d0 con 0x7f73680b7d70 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681606f0 0x7f73680bacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681606f0 0x7f73680bacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56030/0 (socket says 172.21.15.28:56030) 2024-04-25T19:11:25.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 -- 172.21.15.28:0/3049815983 learned_addr learned my addr 172.21.15.28:0/3049815983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f736659c640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73680b7d70 0x7f7368152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f7366d9d640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7368164990 0x7f73680cf350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 -- 172.21.15.28:0/3049815983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73680b7d70 msgr2=0x7f7368152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73680b7d70 0x7f7368152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 -- 172.21.15.28:0/3049815983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7368164990 msgr2=0x7f73680cf350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7368164990 0x7f73680cf350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 -- 172.21.15.28:0/3049815983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73680ca8c0 con 0x7f73681606f0 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735ffff640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681606f0 0x7f73680bacf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7350035040 tx=0x7f73500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735effd640 1 -- 172.21.15.28:0/3049815983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350037050 con 0x7f73681606f0 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735effd640 1 -- 172.21.15.28:0/3049815983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7350038040 con 0x7f73681606f0 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73680caba0 con 0x7f73681606f0 2024-04-25T19:11:25.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.522+0000 7f735effd640 1 -- 172.21.15.28:0/3049815983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7350004470 con 0x7f73681606f0 2024-04-25T19:11:25.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.523+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73680cb160 con 0x7f73681606f0 2024-04-25T19:11:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.524+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7368050740 con 0x7f73681606f0 2024-04-25T19:11:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.524+0000 7f735effd640 1 -- 172.21.15.28:0/3049815983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7350030980 con 0x7f73681606f0 2024-04-25T19:11:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.525+0000 7f735effd640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f73480744b0 0x7f73480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.525+0000 7f735effd640 1 -- 172.21.15.28:0/3049815983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f73500b59a0 con 0x7f73681606f0 2024-04-25T19:11:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.525+0000 7f736659c640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f73480744b0 0x7f73480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.525+0000 7f736659c640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f73480744b0 0x7f73480769c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7358000f60 tx=0x7f7358009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:25.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.530+0000 7f735effd640 1 -- 172.21.15.28:0/3049815983 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f735007fe30 con 0x7f73681606f0 2024-04-25T19:11:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.754+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73680d4000 con 0x7f73681606f0 2024-04-25T19:11:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.755+0000 7f735effd640 1 -- 172.21.15.28:0/3049815983 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3861 (secure 0 0 0) 0x7f7350084130 con 0x7f73681606f0 2024-04-25T19:11:25.757 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:25.757 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T19:11:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f73480744b0 msgr2=0x7f73480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f73480744b0 0x7f73480769c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7358000f60 tx=0x7f7358009040 comp rx=0 tx=0).stop 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681606f0 msgr2=0x7f73680bacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681606f0 0x7f73680bacf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7350035040 tx=0x7f73500046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 shutdown_connections 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f73480744b0 0x7f73480769c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7368164990 0x7f73680cf350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73681606f0 0x7f73680bacf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 --2- 172.21.15.28:0/3049815983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f73680b7d70 0x7f7368152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 >> 172.21.15.28:0/3049815983 conn(0x7f73680d2840 msgr2=0x7f7368162890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 shutdown_connections 2024-04-25T19:11:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.757+0000 7f736cf1f640 1 -- 172.21.15.28:0/3049815983 wait complete. 2024-04-25T19:11:25.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:25.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.879+0000 7fbc9cc82640 1 Processor -- start 2024-04-25T19:11:25.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.880+0000 7fbc9cc82640 1 -- start start 2024-04-25T19:11:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9cc82640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc9806e270 0x7fbc9806e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9cc82640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc980d32c0 con 0x7fbc98165430 2024-04-25T19:11:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9cc82640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc980b9200 con 0x7fbc98164910 2024-04-25T19:11:25.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9cc82640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fbc980d10a0 con 0x7fbc9806e270 2024-04-25T19:11:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9659c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc9806e270 0x7fbc9806e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9659c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc9806e270 0x7fbc9806e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56032/0 (socket says 172.21.15.28:56032) 2024-04-25T19:11:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc96d9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbc98165430 0x7fbc980d1ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53612/0 (socket says 172.21.15.28:53612) 2024-04-25T19:11:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9659c640 1 -- 172.21.15.28:0/2378293218 learned_addr learned my addr 172.21.15.28:0/2378293218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9659c640 1 -- 172.21.15.28:0/2378293218 >> v1:172.21.15.53:6789/0 conn(0x7fbc98164910 legacy=0x7fbc98164d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9659c640 1 -- 172.21.15.28:0/2378293218 >> v1:172.21.15.28:6789/0 conn(0x7fbc98165430 legacy=0x7fbc980d1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9659c640 1 -- 172.21.15.28:0/2378293218 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9809d150 con 0x7fbc9806e270 2024-04-25T19:11:25.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.881+0000 7fbc9559a640 1 -- 172.21.15.28:0/2378293218 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3560303787 0 0) 0x7fbc88001490 con 0x7fbc98165430 2024-04-25T19:11:25.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.882+0000 7fbc9659c640 1 --2- 172.21.15.28:0/2378293218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806e270 0x7fbc9806e680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009870 tx=0x7fbc9002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42a6ca6c9e0a9dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.882+0000 7fbc9559a640 1 -- 172.21.15.28:0/2378293218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90037070 con 0x7fbc9806e270 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.882+0000 7fbc9559a640 1 -- 172.21.15.28:0/2378293218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc90032440 con 0x7fbc9806e270 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.882+0000 7fbc9559a640 1 -- 172.21.15.28:0/2378293218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90032810 con 0x7fbc9806e270 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.883+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2378293218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806e270 msgr2=0x7fbc9806e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.883+0000 7fbc9cc82640 1 --2- 172.21.15.28:0/2378293218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806e270 0x7fbc9806e680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbc90009870 tx=0x7fbc9002a2a0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.883+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2378293218 shutdown_connections 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.883+0000 7fbc9cc82640 1 --2- 172.21.15.28:0/2378293218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806e270 0x7fbc9806e680 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.883+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2378293218 >> 172.21.15.28:0/2378293218 conn(0x7fbc980d03e0 msgr2=0x7fbc980d07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:25.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.883+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2378293218 shutdown_connections 2024-04-25T19:11:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.883+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2378293218 wait complete. 2024-04-25T19:11:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.885+0000 7fbc9cc82640 1 Processor -- start 2024-04-25T19:11:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.885+0000 7fbc9cc82640 1 -- start start 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.886+0000 7fbc9cc82640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9806e270 0x7fbc981f69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.886+0000 7fbc9cc82640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc98164910 0x7fbc980b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.886+0000 7fbc9cc82640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc98165430 0x7fbc980b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.886+0000 7fbc9cc82640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9806d800 con 0x7fbc98164910 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.886+0000 7fbc9cc82640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc9806d680 con 0x7fbc98165430 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.886+0000 7fbc9cc82640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9806ebc0 con 0x7fbc9806e270 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc98164910 0x7fbc980b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc98164910 0x7fbc980b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56034/0 (socket says 172.21.15.28:56034) 2024-04-25T19:11:25.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 -- 172.21.15.28:0/2257470473 learned_addr learned my addr 172.21.15.28:0/2257470473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc96d9d640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc98165430 0x7fbc980b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc9659c640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9806e270 0x7fbc981f69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 -- 172.21.15.28:0/2257470473 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9806e270 msgr2=0x7fbc981f69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9806e270 0x7fbc981f69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 -- 172.21.15.28:0/2257470473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc98165430 msgr2=0x7fbc980b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc98165430 0x7fbc980b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 -- 172.21.15.28:0/2257470473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc98155400 con 0x7fbc98164910 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc95d9b640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc98164910 0x7fbc980b2c90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000b3f0 tx=0x7fbc8000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc8effd640 1 -- 172.21.15.28:0/2257470473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80015020 con 0x7fbc98164910 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc8effd640 1 -- 172.21.15.28:0/2257470473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc80011830 con 0x7fbc98164910 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc980b3710 con 0x7fbc98164910 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc8effd640 1 -- 172.21.15.28:0/2257470473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80011c20 con 0x7fbc98164910 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.887+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc980b3cb0 con 0x7fbc98164910 2024-04-25T19:11:25.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.888+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc98050740 con 0x7fbc98164910 2024-04-25T19:11:25.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.889+0000 7fbc8effd640 1 -- 172.21.15.28:0/2257470473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbc80018020 con 0x7fbc98164910 2024-04-25T19:11:25.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.889+0000 7fbc8effd640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc780744b0 0x7fbc780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:25.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.889+0000 7fbc8effd640 1 -- 172.21.15.28:0/2257470473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbc80094a40 con 0x7fbc98164910 2024-04-25T19:11:25.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.889+0000 7fbc9659c640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc780744b0 0x7fbc780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:25.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.890+0000 7fbc9659c640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc780744b0 0x7fbc780769c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbc90034000 tx=0x7fbc9002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:25.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:25.894+0000 7fbc8effd640 1 -- 172.21.15.28:0/2257470473 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc8001f030 con 0x7fbc98164910 2024-04-25T19:11:26.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.080+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc981ed150 con 0x7fbc98164910 2024-04-25T19:11:26.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.080+0000 7fbc8effd640 1 -- 172.21.15.28:0/2257470473 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12824 (secure 0 0 0) 0x7fbc80094e20 con 0x7fbc98164910 2024-04-25T19:11:26.082 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:26.082 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:18.881227+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc780744b0 msgr2=0x7fbc780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc780744b0 0x7fbc780769c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbc90034000 tx=0x7fbc9002e000 comp rx=0 tx=0).stop 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc98164910 msgr2=0x7fbc980b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc98164910 0x7fbc980b2c90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000b3f0 tx=0x7fbc8000b8b0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 shutdown_connections 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc780744b0 0x7fbc780769c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc98165430 0x7fbc980b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc98164910 0x7fbc980b2c90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 --2- 172.21.15.28:0/2257470473 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9806e270 0x7fbc981f69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.082+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 >> 172.21.15.28:0/2257470473 conn(0x7fbc980d03e0 msgr2=0x7fbc98072690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:26.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.083+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 shutdown_connections 2024-04-25T19:11:26.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.083+0000 7fbc9cc82640 1 -- 172.21.15.28:0/2257470473 wait complete. 2024-04-25T19:11:26.095 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:11:26.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:11:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.209+0000 7fe8cc12e640 1 Processor -- start 2024-04-25T19:11:26.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.209+0000 7fe8cc12e640 1 -- start start 2024-04-25T19:11:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.210+0000 7fe8cc12e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8cc12e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8c4160720 con 0x7fe8c40b9920 2024-04-25T19:11:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8cc12e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8c40d5640 con 0x7fe8c40b79f0 2024-04-25T19:11:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8cc12e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c40d0f40 con 0x7fe8c4160310 2024-04-25T19:11:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8ca6cb640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe8c40b9920 0x7fe8c40b9d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53622/0 (socket says 172.21.15.28:53622) 2024-04-25T19:11:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8ca6cb640 1 -- 172.21.15.28:0/509514455 learned_addr learned my addr 172.21.15.28:0/509514455 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c9eca640 1 --2- 172.21.15.28:0/509514455 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c8ec8640 1 -- 172.21.15.28:0/509514455 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1559112999 0 0) 0x7fe8c4160720 con 0x7fe8c40b9920 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c8ec8640 1 -- 172.21.15.28:0/509514455 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8ac003520 con 0x7fe8c40b9920 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c8ec8640 1 -- 172.21.15.28:0/509514455 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1670610584 0 0) 0x7fe8c40d5640 con 0x7fe8c40b79f0 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c8ec8640 1 -- 172.21.15.28:0/509514455 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8c4160720 con 0x7fe8c40b79f0 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c9eca640 1 -- 172.21.15.28:0/509514455 >> v1:172.21.15.53:6789/0 conn(0x7fe8c40b79f0 legacy=0x7fe8c40b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c9eca640 1 -- 172.21.15.28:0/509514455 >> v1:172.21.15.28:6789/0 conn(0x7fe8c40b9920 legacy=0x7fe8c40b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c8ec8640 1 -- 172.21.15.28:0/509514455 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2218897128 0 0) 0x7fe8ac003520 con 0x7fe8c40b9920 2024-04-25T19:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.211+0000 7fe8c9eca640 1 -- 172.21.15.28:0/509514455 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c40d1220 con 0x7fe8c4160310 2024-04-25T19:11:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.212+0000 7fe8c9eca640 1 --2- 172.21.15.28:0/509514455 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40d5100 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc0097d0 tx=0x7fe8bc02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5ee267f548cd51a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.212+0000 7fe8c8ec8640 1 -- 172.21.15.28:0/509514455 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8bc037070 con 0x7fe8c4160310 2024-04-25T19:11:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.212+0000 7fe8c8ec8640 1 -- 172.21.15.28:0/509514455 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8bc02ac30 con 0x7fe8c4160310 2024-04-25T19:11:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.212+0000 7fe8c8ec8640 1 -- 172.21.15.28:0/509514455 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b40019f0 con 0x7fe8c4160310 2024-04-25T19:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.213+0000 7fe8cc12e640 1 -- 172.21.15.28:0/509514455 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 msgr2=0x7fe8c40d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.213+0000 7fe8cc12e640 1 --2- 172.21.15.28:0/509514455 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40d5100 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc0097d0 tx=0x7fe8bc02a1a0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.213+0000 7fe8cc12e640 1 -- 172.21.15.28:0/509514455 shutdown_connections 2024-04-25T19:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.213+0000 7fe8cc12e640 1 --2- 172.21.15.28:0/509514455 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40d5100 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.213+0000 7fe8cc12e640 1 -- 172.21.15.28:0/509514455 >> 172.21.15.28:0/509514455 conn(0x7fe8c40d2a50 msgr2=0x7fe8c40d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.213+0000 7fe8cc12e640 1 -- 172.21.15.28:0/509514455 shutdown_connections 2024-04-25T19:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.213+0000 7fe8cc12e640 1 -- 172.21.15.28:0/509514455 wait complete. 2024-04-25T19:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.215+0000 7fe8cc12e640 1 Processor -- start 2024-04-25T19:11:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.215+0000 7fe8cc12e640 1 -- start start 2024-04-25T19:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8cc12e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8c40b79f0 0x7fe8c41f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8cc12e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8c40b9920 0x7fe8c4158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8cc12e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40c9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8cc12e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c40d0bf0 con 0x7fe8c40b9920 2024-04-25T19:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8cc12e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c40b9400 con 0x7fe8c40b79f0 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8cc12e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c40cc950 con 0x7fe8c4160310 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8c96c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8c40b9920 0x7fe8c4158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8c96c9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8c40b9920 0x7fe8c4158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56048/0 (socket says 172.21.15.28:56048) 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8c96c9640 1 -- 172.21.15.28:0/333450850 learned_addr learned my addr 172.21.15.28:0/333450850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8c9eca640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8c40b79f0 0x7fe8c41f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.216+0000 7fe8ca6cb640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c96c9640 1 -- 172.21.15.28:0/333450850 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 msgr2=0x7fe8c40c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c96c9640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40c9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c96c9640 1 -- 172.21.15.28:0/333450850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8c40b79f0 msgr2=0x7fe8c41f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c96c9640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8c40b79f0 0x7fe8c41f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c96c9640 1 -- 172.21.15.28:0/333450850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c4159000 con 0x7fe8c40b9920 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c96c9640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8c40b9920 0x7fe8c4158290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800c3c0 tx=0x7fe8b800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c27fc640 1 -- 172.21.15.28:0/333450850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8017070 con 0x7fe8c40b9920 2024-04-25T19:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c27fc640 1 -- 172.21.15.28:0/333450850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b80052f0 con 0x7fe8c40b9920 2024-04-25T19:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c41592e0 con 0x7fe8c40b9920 2024-04-25T19:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8c27fc640 1 -- 172.21.15.28:0/333450850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b8003830 con 0x7fe8c40b9920 2024-04-25T19:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.217+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c41598a0 con 0x7fe8c40b9920 2024-04-25T19:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.218+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8c40d5630 con 0x7fe8c40b9920 2024-04-25T19:11:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.219+0000 7fe8c27fc640 1 -- 172.21.15.28:0/333450850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe8b801a070 con 0x7fe8c40b9920 2024-04-25T19:11:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.219+0000 7fe8c27fc640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe8b00744b0 0x7fe8b00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.219+0000 7fe8c9eca640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe8b00744b0 0x7fe8b00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.219+0000 7fe8c27fc640 1 -- 172.21.15.28:0/333450850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe8b80941f0 con 0x7fe8c40b9920 2024-04-25T19:11:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.220+0000 7fe8c9eca640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe8b00744b0 0x7fe8b00769c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc034000 tx=0x7fe8bc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.224+0000 7fe8c27fc640 1 -- 172.21.15.28:0/333450850 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8c40d5630 con 0x7fe8c40b9920 2024-04-25T19:11:26.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.408+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8c4158d90 con 0x7fe8c40b9920 2024-04-25T19:11:26.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.409+0000 7fe8c27fc640 1 -- 172.21.15.28:0/333450850 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7fe8b805e740 con 0x7fe8c40b9920 2024-04-25T19:11:26.411 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe8b00744b0 msgr2=0x7fe8b00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe8b00744b0 0x7fe8b00769c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc034000 tx=0x7fe8bc02e000 comp rx=0 tx=0).stop 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8c40b9920 msgr2=0x7fe8c4158290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8c40b9920 0x7fe8c4158290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe8b800c3c0 tx=0x7fe8b800c880 comp rx=0 tx=0).stop 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 shutdown_connections 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe8b00744b0 0x7fe8b00769c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe8c4160310 0x7fe8c40c9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe8c40b9920 0x7fe8c4158290 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 --2- 172.21.15.28:0/333450850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8c40b79f0 0x7fe8c41f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 >> 172.21.15.28:0/333450850 conn(0x7fe8c40d2a50 msgr2=0x7fe8c415f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 shutdown_connections 2024-04-25T19:11:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.411+0000 7fe8cc12e640 1 -- 172.21.15.28:0/333450850 wait complete. 2024-04-25T19:11:26.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T19:11:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.533+0000 7f91f0813640 1 Processor -- start 2024-04-25T19:11:26.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.533+0000 7f91f0813640 1 -- start start 2024-04-25T19:11:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91f0813640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b4e30 0x7f91ec0bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91f0813640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f91ec0b9470 0x7f91ec0b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91f0813640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91ec0bbb90 con 0x7f91ec0b7650 2024-04-25T19:11:26.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91f0813640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f91ec0d0b50 con 0x7f91ec0b9470 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91f0813640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f91ec08e3a0 con 0x7f91ec0b4e30 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b4e30 0x7f91ec0bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91e3fff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f91ec0b9470 0x7f91ec0b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b4e30 0x7f91ec0bb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:57944/0 (socket says 172.21.15.28:57944) 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91ead9d640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f91ec0b7650 0x7f91ec0b7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:47076/0 (socket says 172.21.15.28:47076) 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91ea59c640 1 -- 172.21.15.28:0/1379514878 learned_addr learned my addr 172.21.15.28:0/1379514878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91ea59c640 1 -- 172.21.15.28:0/1379514878 >> v2:172.21.15.53:3300/0 conn(0x7f91ec0b9470 msgr2=0x7f91ec0b9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91ea59c640 1 --2- 172.21.15.28:0/1379514878 >> v2:172.21.15.53:3300/0 conn(0x7f91ec0b9470 0x7f91ec0b9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91ea59c640 1 -- 172.21.15.28:0/1379514878 >> v1:172.21.15.53:6789/0 conn(0x7f91ec0b7650 legacy=0x7f91ec0b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.535+0000 7f91ea59c640 1 -- 172.21.15.28:0/1379514878 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec164d60 con 0x7f91ec0b4e30 2024-04-25T19:11:26.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.536+0000 7f91ea59c640 1 --2- 172.21.15.28:0/1379514878 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b4e30 0x7f91ec0bb650 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009870 tx=0x7f91dc02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25792bacd6608f19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.536+0000 7f91e9d9b640 1 -- 172.21.15.28:0/1379514878 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc037070 con 0x7f91ec0b4e30 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.536+0000 7f91e9d9b640 1 -- 172.21.15.28:0/1379514878 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91dc032440 con 0x7f91ec0b4e30 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91e9d9b640 1 -- 172.21.15.28:0/1379514878 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc032810 con 0x7f91ec0b4e30 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91f0813640 1 -- 172.21.15.28:0/1379514878 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b4e30 msgr2=0x7f91ec0bb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91f0813640 1 --2- 172.21.15.28:0/1379514878 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b4e30 0x7f91ec0bb650 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f91dc009870 tx=0x7f91dc02a1c0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91f0813640 1 -- 172.21.15.28:0/1379514878 shutdown_connections 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91f0813640 1 --2- 172.21.15.28:0/1379514878 >> v2:172.21.15.53:3300/0 conn(0x7f91ec0b9470 0x7f91ec0b9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91f0813640 1 --2- 172.21.15.28:0/1379514878 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b4e30 0x7f91ec0bb650 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91f0813640 1 -- 172.21.15.28:0/1379514878 >> 172.21.15.28:0/1379514878 conn(0x7f91ec0d2650 msgr2=0x7f91ec0d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91f0813640 1 -- 172.21.15.28:0/1379514878 shutdown_connections 2024-04-25T19:11:26.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.537+0000 7f91f0813640 1 -- 172.21.15.28:0/1379514878 wait complete. 2024-04-25T19:11:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.539+0000 7f91f0813640 1 Processor -- start 2024-04-25T19:11:26.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.539+0000 7f91f0813640 1 -- start start 2024-04-25T19:11:26.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.540+0000 7f91f0813640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec0b4e30 0x7f91ec14a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.540+0000 7f91f0813640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ec0b7650 0x7f91ec0cf700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.540+0000 7f91f0813640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b9470 0x7f91ec162050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.540+0000 7f91f0813640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec0d07e0 con 0x7f91ec0b4e30 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.540+0000 7f91f0813640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f91ec0d0500 con 0x7f91ec0b7650 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.540+0000 7f91f0813640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f91ec1649c0 con 0x7f91ec0b9470 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec0b4e30 0x7f91ec14a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec0b4e30 0x7f91ec14a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56064/0 (socket says 172.21.15.28:56064) 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 -- 172.21.15.28:0/133797541 learned_addr learned my addr 172.21.15.28:0/133797541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ead9d640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b9470 0x7f91ec162050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91e3fff640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ec0b7650 0x7f91ec0cf700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 -- 172.21.15.28:0/133797541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b9470 msgr2=0x7f91ec162050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b9470 0x7f91ec162050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 -- 172.21.15.28:0/133797541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ec0b7650 msgr2=0x7f91ec0cf700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ec0b7650 0x7f91ec0cf700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 -- 172.21.15.28:0/133797541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ec1eacf0 con 0x7f91ec0b4e30 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91ea59c640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec0b4e30 0x7f91ec14a030 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91dc000c00 tx=0x7f91dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91e2ffd640 1 -- 172.21.15.28:0/133797541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc037040 con 0x7f91ec0b4e30 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91e2ffd640 1 -- 172.21.15.28:0/133797541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91dc042070 con 0x7f91ec0b4e30 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ec1ea450 con 0x7f91ec0b4e30 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91e2ffd640 1 -- 172.21.15.28:0/133797541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc030860 con 0x7f91ec0b4e30 2024-04-25T19:11:26.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.541+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ec1eaa10 con 0x7f91ec0b4e30 2024-04-25T19:11:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.543+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ec050740 con 0x7f91ec0b4e30 2024-04-25T19:11:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.543+0000 7f91e2ffd640 1 -- 172.21.15.28:0/133797541 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f91dc02ad10 con 0x7f91ec0b4e30 2024-04-25T19:11:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.543+0000 7f91e2ffd640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91cc0744b0 0x7f91cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.543+0000 7f91e2ffd640 1 -- 172.21.15.28:0/133797541 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f91dc0b5380 con 0x7f91ec0b4e30 2024-04-25T19:11:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.544+0000 7f91e3fff640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91cc0744b0 0x7f91cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.544+0000 7f91e3fff640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91cc0744b0 0x7f91cc0769c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f91ec155fa0 tx=0x7f91d4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.548+0000 7f91e2ffd640 1 -- 172.21.15.28:0/133797541 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91dc07f800 con 0x7f91ec0b4e30 2024-04-25T19:11:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.739+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f91ec0b8ab0 con 0x7f91ec0b4e30 2024-04-25T19:11:26.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.740+0000 7f91e2ffd640 1 -- 172.21.15.28:0/133797541 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f91dc083b00 con 0x7f91ec0b4e30 2024-04-25T19:11:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:26.742 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-04-25T19:11:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:11:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-04-25T19:11:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-04-25T19:11:26.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:11:26.742 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91cc0744b0 msgr2=0x7f91cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91cc0744b0 0x7f91cc0769c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f91ec155fa0 tx=0x7f91d4009190 comp rx=0 tx=0).stop 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec0b4e30 msgr2=0x7f91ec14a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec0b4e30 0x7f91ec14a030 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f91dc000c00 tx=0x7f91dc0046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 shutdown_connections 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91cc0744b0 0x7f91cc0769c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91ec0b9470 0x7f91ec162050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91ec0b7650 0x7f91ec0cf700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 --2- 172.21.15.28:0/133797541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91ec0b4e30 0x7f91ec14a030 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 >> 172.21.15.28:0/133797541 conn(0x7f91ec0d2650 msgr2=0x7f91ec0b9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 shutdown_connections 2024-04-25T19:11:26.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.742+0000 7f91f0813640 1 -- 172.21.15.28:0/133797541 wait complete. 2024-04-25T19:11:26.755 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T19:11:26.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T19:11:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.868+0000 7fb703b2d640 1 Processor -- start 2024-04-25T19:11:26.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.869+0000 7fb703b2d640 1 -- start start 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb703b2d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6fc0d5690 0x7fb6fc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb703b2d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0b4d00 0x7fb6fc06e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb703b2d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6fc0d12e0 con 0x7fb6fc0b47d0 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb703b2d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb6fc0d15c0 con 0x7fb6fc0d5690 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb703b2d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb6fc0d4bb0 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7010c8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6fc0d5690 0x7fb6fc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7010c8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6fc0d5690 0x7fb6fc06ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56080/0 (socket says 172.21.15.28:56080) 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7010c8640 1 -- 172.21.15.28:0/50070423 learned_addr learned my addr 172.21.15.28:0/50070423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7020ca640 1 --2- 172.21.15.28:0/50070423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0b4d00 0x7fb6fc06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7010c8640 1 -- 172.21.15.28:0/50070423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0b4d00 msgr2=0x7fb6fc06e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7010c8640 1 --2- 172.21.15.28:0/50070423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0b4d00 0x7fb6fc06e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7010c8640 1 -- 172.21.15.28:0/50070423 >> v1:172.21.15.53:6789/0 conn(0x7fb6fc0b47d0 legacy=0x7fb6fc1601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7010c8640 1 -- 172.21.15.28:0/50070423 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc0d4e90 con 0x7fb6fc0d5690 2024-04-25T19:11:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.870+0000 7fb7020ca640 1 --2- 172.21.15.28:0/50070423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0b4d00 0x7fb6fc06e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.871+0000 7fb7010c8640 1 --2- 172.21.15.28:0/50070423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0d5690 0x7fb6fc06ded0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009870 tx=0x7fb6f002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5d70c017b579f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.871+0000 7fb7008c7640 1 -- 172.21.15.28:0/50070423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0037070 con 0x7fb6fc0d5690 2024-04-25T19:11:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.871+0000 7fb7008c7640 1 -- 172.21.15.28:0/50070423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f0032440 con 0x7fb6fc0d5690 2024-04-25T19:11:26.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.871+0000 7fb7008c7640 1 -- 172.21.15.28:0/50070423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0032810 con 0x7fb6fc0d5690 2024-04-25T19:11:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.872+0000 7fb703b2d640 1 -- 172.21.15.28:0/50070423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0d5690 msgr2=0x7fb6fc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.872+0000 7fb703b2d640 1 --2- 172.21.15.28:0/50070423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0d5690 0x7fb6fc06ded0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009870 tx=0x7fb6f002a270 comp rx=0 tx=0).stop 2024-04-25T19:11:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.872+0000 7fb703b2d640 1 -- 172.21.15.28:0/50070423 shutdown_connections 2024-04-25T19:11:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.872+0000 7fb703b2d640 1 --2- 172.21.15.28:0/50070423 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0b4d00 0x7fb6fc06e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.872+0000 7fb703b2d640 1 --2- 172.21.15.28:0/50070423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0d5690 0x7fb6fc06ded0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.872+0000 7fb703b2d640 1 -- 172.21.15.28:0/50070423 >> 172.21.15.28:0/50070423 conn(0x7fb6fc0d2c60 msgr2=0x7fb6fc0d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.872+0000 7fb703b2d640 1 -- 172.21.15.28:0/50070423 shutdown_connections 2024-04-25T19:11:26.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.872+0000 7fb703b2d640 1 -- 172.21.15.28:0/50070423 wait complete. 2024-04-25T19:11:26.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.874+0000 7fb703b2d640 1 Processor -- start 2024-04-25T19:11:26.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.874+0000 7fb703b2d640 1 -- start start 2024-04-25T19:11:26.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb703b2d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6fc0b47d0 0x7fb6fc0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb703b2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0b4d00 0x7fb6fc15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb703b2d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0d5690 0x7fb6fc0cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb703b2d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc0b9380 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb703b2d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6fc0b9200 con 0x7fb6fc0b47d0 2024-04-25T19:11:26.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb703b2d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb6fc0b9500 con 0x7fb6fc0d5690 2024-04-25T19:11:26.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb7010c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0b4d00 0x7fb6fc15c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb7010c8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0b4d00 0x7fb6fc15c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56096/0 (socket says 172.21.15.28:56096) 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb7010c8640 1 -- 172.21.15.28:0/1127175714 learned_addr learned my addr 172.21.15.28:0/1127175714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb7020ca640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0d5690 0x7fb6fc0cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb7010c8640 1 -- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0d5690 msgr2=0x7fb6fc0cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb7018c9640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6fc0b47d0 0x7fb6fc0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb7010c8640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0d5690 0x7fb6fc0cad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.875+0000 7fb7010c8640 1 -- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6fc0b47d0 msgr2=0x7fb6fc0ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb7010c8640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6fc0b47d0 0x7fb6fc0ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb7010c8640 1 -- 172.21.15.28:0/1127175714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6fc15ca40 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb7018c9640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6fc0b47d0 0x7fb6fc0ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb7010c8640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0b4d00 0x7fb6fc15c500 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009870 tx=0x7fb6f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb6fa7fc640 1 -- 172.21.15.28:0/1127175714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0041020 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb6fa7fc640 1 -- 172.21.15.28:0/1127175714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6f00306c0 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6fc162cc0 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb6fa7fc640 1 -- 172.21.15.28:0/1127175714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0037040 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.876+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6fc0ca530 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.877+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6fc0ca080 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.877+0000 7fb6fa7fc640 1 -- 172.21.15.28:0/1127175714 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb6f0044020 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.878+0000 7fb6fa7fc640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6e80744b0 0x7fb6e80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.878+0000 7fb7018c9640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6e80744b0 0x7fb6e80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.878+0000 7fb6fa7fc640 1 -- 172.21.15.28:0/1127175714 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fb6f00b57b0 con 0x7fb6fc0b4d00 2024-04-25T19:11:26.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.879+0000 7fb7018c9640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6e80744b0 0x7fb6e80769c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4000f60 tx=0x7fb6f4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:26.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:26.883+0000 7fb6fa7fc640 1 -- 172.21.15.28:0/1127175714 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6f004b030 con 0x7fb6fc0b4d00 2024-04-25T19:11:27.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.068+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fb6fc0d06c0 con 0x7fb6fc0b4d00 2024-04-25T19:11:27.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.906+0000 7fb6fa7fc640 1 -- 172.21.15.28:0/1127175714 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fb6fc0d06c0 con 0x7fb6fc0b4d00 2024-04-25T19:11:27.908 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6e80744b0 msgr2=0x7fb6e80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6e80744b0 0x7fb6e80769c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4000f60 tx=0x7fb6f4009040 comp rx=0 tx=0).stop 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0b4d00 msgr2=0x7fb6fc15c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0b4d00 0x7fb6fc15c500 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009870 tx=0x7fb6f00046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 shutdown_connections 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6e80744b0 0x7fb6e80769c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6fc0d5690 0x7fb6fc0cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc0b4d00 0x7fb6fc15c500 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 --2- 172.21.15.28:0/1127175714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6fc0b47d0 0x7fb6fc0ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 >> 172.21.15.28:0/1127175714 conn(0x7fb6fc0d2c60 msgr2=0x7fb6fc15e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:27.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 shutdown_connections 2024-04-25T19:11:27.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:27.911+0000 7fb703b2d640 1 -- 172.21.15.28:0/1127175714 wait complete. 2024-04-25T19:11:27.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T19:11:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.052+0000 7f3885127640 1 Processor -- start 2024-04-25T19:11:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.052+0000 7f3885127640 1 -- start start 2024-04-25T19:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f3885127640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f38800d25b0 0x7f38800d29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f3885127640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38800d2f00 con 0x7f38800b4e10 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f3885127640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3880161150 con 0x7f38800d4480 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f3885127640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f3880161430 con 0x7f38800d25b0 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387ed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f38800b4e10 0x7f3880164650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53638/0 (socket says 172.21.15.28:53638) 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387ed9d640 1 -- 172.21.15.28:0/925684962 learned_addr learned my addr 172.21.15.28:0/925684962 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387e59c640 1 --2- 172.21.15.28:0/925684962 >> v2:172.21.15.53:3300/0 conn(0x7f38800d25b0 0x7f38800d29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464748693 0 0) 0x7f3878001110 con 0x7f38800b4e10 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3868003520 con 0x7f38800b4e10 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1748409767 0 0) 0x7f3868003520 con 0x7f38800b4e10 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 >> v2:172.21.15.53:3300/0 conn(0x7f38800d25b0 msgr2=0x7f38800d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 --2- 172.21.15.28:0/925684962 >> v2:172.21.15.53:3300/0 conn(0x7f38800d25b0 0x7f38800d29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 >> v1:172.21.15.53:6789/0 conn(0x7f38800d4480 legacy=0x7f38800d4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3880163e50 con 0x7f38800b4e10 2024-04-25T19:11:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f3878001840 con 0x7f38800b4e10 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 >> v1:172.21.15.28:6789/0 conn(0x7f38800b4e10 legacy=0x7f3880164650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38680039b0 0x7f3868003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868004800 0x7f3868004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868005250 0x7f386800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3880161430 con 0x7f3868005250 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f386800c1e0 con 0x7f38680039b0 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f386800c3e0 con 0x7f3868004800 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.054+0000 7f387f59e640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868005250 0x7f386800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387ed9d640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38680039b0 0x7f3868003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387e59c640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868004800 0x7f3868004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387f59e640 1 -- 172.21.15.28:0/925684962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868004800 msgr2=0x7f3868004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387f59e640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868004800 0x7f3868004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387f59e640 1 -- 172.21.15.28:0/925684962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38680039b0 msgr2=0x7f3868003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387f59e640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38680039b0 0x7f3868003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387f59e640 1 -- 172.21.15.28:0/925684962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3868003520 con 0x7f3868005250 2024-04-25T19:11:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387f59e640 1 -- 172.21.15.28:0/925684962 reap_dead start 2024-04-25T19:11:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.055+0000 7f387f59e640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868005250 0x7f386800b9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3870001e20 tx=0x7f3868005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aba84852bbc02c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870001a70 con 0x7f3868005250 2024-04-25T19:11:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f387002ae40 con 0x7f3868005250 2024-04-25T19:11:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f387dd9b640 1 -- 172.21.15.28:0/925684962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3870003b90 con 0x7f3868005250 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f3885127640 1 -- 172.21.15.28:0/925684962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868005250 msgr2=0x7f386800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f3885127640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868005250 0x7f386800b9c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3870001e20 tx=0x7f3868005830 comp rx=0 tx=0).stop 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f3885127640 1 -- 172.21.15.28:0/925684962 shutdown_connections 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f3885127640 1 --2- 172.21.15.28:0/925684962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3868005250 0x7f386800b9c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f3885127640 1 -- 172.21.15.28:0/925684962 >> 172.21.15.28:0/925684962 conn(0x7f388006ee10 msgr2=0x7f388006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f3885127640 1 -- 172.21.15.28:0/925684962 shutdown_connections 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.056+0000 7f3885127640 1 -- 172.21.15.28:0/925684962 wait complete. 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.058+0000 7f3885127640 1 Processor -- start 2024-04-25T19:11:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.058+0000 7f3885127640 1 -- start start 2024-04-25T19:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f3885127640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868005250 0x7f38800c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f3885127640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388015d100 0x7f38800cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f3885127640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38800cb310 0x7f38801f9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f3885127640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38800d29c0 con 0x7f388015d100 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f3885127640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f38800b7850 con 0x7f38800cb310 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f3885127640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f38800d2b40 con 0x7f3868005250 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388015d100 0x7f38800cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388015d100 0x7f38800cadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56110/0 (socket says 172.21.15.28:56110) 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 -- 172.21.15.28:0/1447124460 learned_addr learned my addr 172.21.15.28:0/1447124460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387ed9d640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868005250 0x7f38800c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387f59e640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38800cb310 0x7f38801f9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 -- 172.21.15.28:0/1447124460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868005250 msgr2=0x7f38800c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868005250 0x7f38800c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 -- 172.21.15.28:0/1447124460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38800cb310 msgr2=0x7f38801f9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38800cb310 0x7f38801f9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 -- 172.21.15.28:0/1447124460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38800b66a0 con 0x7f388015d100 2024-04-25T19:11:28.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.060+0000 7f387e59c640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388015d100 0x7f38800cadd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3874011960 tx=0x7f3874011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:28.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.061+0000 7f38677fe640 1 -- 172.21.15.28:0/1447124460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3874019070 con 0x7f388015d100 2024-04-25T19:11:28.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.061+0000 7f38677fe640 1 -- 172.21.15.28:0/1447124460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3874004240 con 0x7f388015d100 2024-04-25T19:11:28.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.061+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38800c9d30 con 0x7f388015d100 2024-04-25T19:11:28.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.061+0000 7f38677fe640 1 -- 172.21.15.28:0/1447124460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387400f820 con 0x7f388015d100 2024-04-25T19:11:28.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.061+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38800ca2f0 con 0x7f388015d100 2024-04-25T19:11:28.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.062+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3880050740 con 0x7f388015d100 2024-04-25T19:11:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.063+0000 7f38677fe640 1 -- 172.21.15.28:0/1447124460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f387401c070 con 0x7f388015d100 2024-04-25T19:11:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.064+0000 7f38677fe640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f386c0744e0 0x7f386c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.064+0000 7f38677fe640 1 -- 172.21.15.28:0/1447124460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f38740955f0 con 0x7f388015d100 2024-04-25T19:11:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.064+0000 7f387ed9d640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f386c0744e0 0x7f386c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:28.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.064+0000 7f387ed9d640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f386c0744e0 0x7f386c0769f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3878001eb0 tx=0x7f38780012b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:28.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.068+0000 7f38677fe640 1 -- 172.21.15.28:0/1447124460 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38800b66a0 con 0x7f388015d100 2024-04-25T19:11:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.253+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f38800bb700 con 0x7f388015d100 2024-04-25T19:11:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.912+0000 7f38677fe640 1 -- 172.21.15.28:0/1447124460 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7f387405fb10 con 0x7f388015d100 2024-04-25T19:11:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-04-25T19:11:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.915+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f386c0744e0 msgr2=0x7f386c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.915+0000 7f3885127640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f386c0744e0 0x7f386c0769f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3878001eb0 tx=0x7f38780012b0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.915+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388015d100 msgr2=0x7f38800cadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.915+0000 7f3885127640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388015d100 0x7f38800cadd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3874011960 tx=0x7f3874011d60 comp rx=0 tx=0).stop 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.916+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 shutdown_connections 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.916+0000 7f3885127640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f386c0744e0 0x7f386c0769f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.916+0000 7f3885127640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38800cb310 0x7f38801f9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.916+0000 7f3885127640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f388015d100 0x7f38800cadd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.916+0000 7f3885127640 1 --2- 172.21.15.28:0/1447124460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3868005250 0x7f38800c9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.916+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 >> 172.21.15.28:0/1447124460 conn(0x7f388006ee10 msgr2=0x7f3880161f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.916+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 shutdown_connections 2024-04-25T19:11:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:28.916+0000 7f3885127640 1 -- 172.21.15.28:0/1447124460 wait complete. 2024-04-25T19:11:28.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T19:11:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.065+0000 7f205510d640 1 Processor -- start 2024-04-25T19:11:29.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.065+0000 7f205510d640 1 -- start start 2024-04-25T19:11:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f205510d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f205015dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f205510d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f20500b7660 0x7f20500b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:29.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f205510d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f20501606f0 0x7f2050160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f205510d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f20500b7fb0 con 0x7f20500b7660 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f205510d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f20501614a0 con 0x7f20501606f0 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f205510d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2050163eb0 con 0x7f2050164990 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f20500b7660 0x7f20500b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f20500b7660 0x7f20500b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56112/0 (socket says 172.21.15.28:56112) 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204e59c640 1 -- 172.21.15.28:0/3899943416 learned_addr learned my addr 172.21.15.28:0/3899943416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204ed9d640 1 --2- 172.21.15.28:0/3899943416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f205015dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204f59e640 1 --2- 172.21.15.28:0/3899943416 >> v2:172.21.15.53:3300/0 conn(0x7f20501606f0 0x7f2050160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204e59c640 1 -- 172.21.15.28:0/3899943416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 msgr2=0x7f205015dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204e59c640 1 --2- 172.21.15.28:0/3899943416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f205015dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204e59c640 1 -- 172.21.15.28:0/3899943416 >> v2:172.21.15.53:3300/0 conn(0x7f20501606f0 msgr2=0x7f2050160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:29.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204e59c640 1 --2- 172.21.15.28:0/3899943416 >> v2:172.21.15.53:3300/0 conn(0x7f20501606f0 0x7f2050160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204e59c640 1 -- 172.21.15.28:0/3899943416 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2050164190 con 0x7f20500b7660 2024-04-25T19:11:29.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.067+0000 7f204f59e640 1 --2- 172.21.15.28:0/3899943416 >> v2:172.21.15.53:3300/0 conn(0x7f20501606f0 0x7f2050160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.068+0000 7f204e59c640 1 --2- 172.21.15.28:0/3899943416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20500b7660 0x7f20500b7a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f20500b86a0 tx=0x7f204402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ec475152cec008 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.068+0000 7f204dd9b640 1 -- 172.21.15.28:0/3899943416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044037070 con 0x7f20500b7660 2024-04-25T19:11:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.068+0000 7f204dd9b640 1 -- 172.21.15.28:0/3899943416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2044032440 con 0x7f20500b7660 2024-04-25T19:11:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.068+0000 7f204dd9b640 1 -- 172.21.15.28:0/3899943416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044032810 con 0x7f20500b7660 2024-04-25T19:11:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 -- 172.21.15.28:0/3899943416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20500b7660 msgr2=0x7f20500b7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 --2- 172.21.15.28:0/3899943416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20500b7660 0x7f20500b7a70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f20500b86a0 tx=0x7f204402a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 -- 172.21.15.28:0/3899943416 shutdown_connections 2024-04-25T19:11:29.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 --2- 172.21.15.28:0/3899943416 >> v2:172.21.15.53:3300/0 conn(0x7f20501606f0 0x7f2050160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 --2- 172.21.15.28:0/3899943416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20500b7660 0x7f20500b7a70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 --2- 172.21.15.28:0/3899943416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f205015dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 -- 172.21.15.28:0/3899943416 >> 172.21.15.28:0/3899943416 conn(0x7f20500d2840 msgr2=0x7f20500d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 -- 172.21.15.28:0/3899943416 shutdown_connections 2024-04-25T19:11:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.069+0000 7f205510d640 1 -- 172.21.15.28:0/3899943416 wait complete. 2024-04-25T19:11:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.071+0000 7f205510d640 1 Processor -- start 2024-04-25T19:11:29.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.071+0000 7f205510d640 1 -- start start 2024-04-25T19:11:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.072+0000 7f205510d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20500b7660 0x7f2050150090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.072+0000 7f205510d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20501606f0 0x7f20501f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.072+0000 7f205510d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f20501ea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:29.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.072+0000 7f205510d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20501611c0 con 0x7f20501606f0 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.072+0000 7f205510d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2050161040 con 0x7f20500b7660 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.072+0000 7f205510d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f20500b45d0 con 0x7f2050164990 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.072+0000 7f204e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20501606f0 0x7f20501f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20501606f0 0x7f20501f61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56120/0 (socket says 172.21.15.28:56120) 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204e59c640 1 -- 172.21.15.28:0/118108753 learned_addr learned my addr 172.21.15.28:0/118108753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204ed9d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20500b7660 0x7f2050150090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204f59e640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f20501ea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204e59c640 1 -- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 msgr2=0x7f20501ea1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204e59c640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f20501ea1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204e59c640 1 -- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20500b7660 msgr2=0x7f2050150090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204e59c640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20500b7660 0x7f2050150090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204e59c640 1 -- 172.21.15.28:0/118108753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20501f6c60 con 0x7f20501606f0 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204f59e640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f20501ea1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f204e59c640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20501606f0 0x7f20501f61e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2044000c00 tx=0x7f20440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:29.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f20377fe640 1 -- 172.21.15.28:0/118108753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044041020 con 0x7f20501606f0 2024-04-25T19:11:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f20377fe640 1 -- 172.21.15.28:0/118108753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20440306c0 con 0x7f20501606f0 2024-04-25T19:11:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20501f6f40 con 0x7f20501606f0 2024-04-25T19:11:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f20377fe640 1 -- 172.21.15.28:0/118108753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2044037040 con 0x7f20501606f0 2024-04-25T19:11:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.073+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20501f6720 con 0x7f20501606f0 2024-04-25T19:11:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.074+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20501f69d0 con 0x7f20501606f0 2024-04-25T19:11:29.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.075+0000 7f20377fe640 1 -- 172.21.15.28:0/118108753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2044044020 con 0x7f20501606f0 2024-04-25T19:11:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.075+0000 7f20377fe640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f203c0744b0 0x7f203c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.076+0000 7f204ed9d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f203c0744b0 0x7f203c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.076+0000 7f20377fe640 1 -- 172.21.15.28:0/118108753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f20440b5980 con 0x7f20501606f0 2024-04-25T19:11:29.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.076+0000 7f204ed9d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f203c0744b0 0x7f203c0769c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2048006fd0 tx=0x7f2048008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:29.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.080+0000 7f20377fe640 1 -- 172.21.15.28:0/118108753 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f204404b030 con 0x7f20501606f0 2024-04-25T19:11:29.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.307+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f2050050740 con 0x7f20501606f0 2024-04-25T19:11:29.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.914+0000 7f20377fe640 1 -- 172.21.15.28:0/118108753 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f2044080200 con 0x7f20501606f0 2024-04-25T19:11:29.916 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T19:11:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.918+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f203c0744b0 msgr2=0x7f203c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.918+0000 7f205510d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f203c0744b0 0x7f203c0769c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2048006fd0 tx=0x7f2048008040 comp rx=0 tx=0).stop 2024-04-25T19:11:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.918+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20501606f0 msgr2=0x7f20501f61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.918+0000 7f205510d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20501606f0 0x7f20501f61e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2044000c00 tx=0x7f20440046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.919+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 shutdown_connections 2024-04-25T19:11:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.919+0000 7f205510d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f203c0744b0 0x7f203c0769c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.919+0000 7f205510d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2050164990 0x7f20501ea1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.919+0000 7f205510d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20501606f0 0x7f20501f61e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.919+0000 7f205510d640 1 --2- 172.21.15.28:0/118108753 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20500b7660 0x7f2050150090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:29.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.920+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 >> 172.21.15.28:0/118108753 conn(0x7f20500d2840 msgr2=0x7f20500d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.922+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 shutdown_connections 2024-04-25T19:11:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:29.923+0000 7f205510d640 1 -- 172.21.15.28:0/118108753 wait complete. 2024-04-25T19:11:29.956 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T19:11:29.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T19:11:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.076+0000 7fcf50884640 1 Processor -- start 2024-04-25T19:11:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.076+0000 7fcf50884640 1 -- start start 2024-04-25T19:11:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.077+0000 7fcf50884640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf4c06e5e0 0x7fcf4c16aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.077+0000 7fcf50884640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf4c0d12e0 con 0x7fcf4c0b47d0 2024-04-25T19:11:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.077+0000 7fcf50884640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf4c0d15c0 con 0x7fcf4c0d5690 2024-04-25T19:11:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.077+0000 7fcf50884640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4c0d4bb0 con 0x7fcf4c06e5e0 2024-04-25T19:11:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.077+0000 7fcf4a59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcf4c0b47d0 0x7fcf4c1601f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53646/0 (socket says 172.21.15.28:53646) 2024-04-25T19:11:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.077+0000 7fcf4a59c640 1 -- 172.21.15.28:0/3295408706 learned_addr learned my addr 172.21.15.28:0/3295408706 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf4ad9d640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf4c06e5e0 0x7fcf4c16aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:30.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367124957 0 0) 0x7fcf4c0d12e0 con 0x7fcf4c0b47d0 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf38003520 con 0x7fcf4c0b47d0 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872180273 0 0) 0x7fcf4c0d15c0 con 0x7fcf4c0d5690 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf4c0d12e0 con 0x7fcf4c0d5690 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3393381172 0 0) 0x7fcf38003520 con 0x7fcf4c0b47d0 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf4c06e5e0 msgr2=0x7fcf4c16aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf4c06e5e0 0x7fcf4c16aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 >> v1:172.21.15.53:6789/0 conn(0x7fcf4c0d5690 legacy=0x7fcf4c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c0d4e90 con 0x7fcf4c0b47d0 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fcf3c001690 con 0x7fcf4c0b47d0 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 >> v1:172.21.15.28:6789/0 conn(0x7fcf4c0b47d0 legacy=0x7fcf4c1601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38003b70 0x7fcf38003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf38004800 0x7fcf38004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf38005150 0x7fcf3800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:30.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c0d4bb0 con 0x7fcf38004800 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcf3800bfb0 con 0x7fcf38005150 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcf3800c1b0 con 0x7fcf38003b70 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf43fff640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf38004800 0x7fcf38004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf4a59c640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38003b70 0x7fcf38003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.078+0000 7fcf4ad9d640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf38005150 0x7fcf3800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf43fff640 1 -- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38003b70 msgr2=0x7fcf38003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf43fff640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38003b70 0x7fcf38003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf43fff640 1 -- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf38005150 msgr2=0x7fcf3800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf43fff640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf38005150 0x7fcf3800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf43fff640 1 -- 172.21.15.28:0/3295408706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf38003520 con 0x7fcf38004800 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf4ad9d640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf38005150 0x7fcf3800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf43fff640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf38004800 0x7fcf38004c10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf340018d0 tx=0x7fcf38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4493912d562a2bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf340014f0 con 0x7fcf38004800 2024-04-25T19:11:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf340052d0 con 0x7fcf38004800 2024-04-25T19:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.079+0000 7fcf49d9b640 1 -- 172.21.15.28:0/3295408706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3402aca0 con 0x7fcf38004800 2024-04-25T19:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.080+0000 7fcf50884640 1 -- 172.21.15.28:0/3295408706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf38004800 msgr2=0x7fcf38004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.080+0000 7fcf50884640 1 --2- 172.21.15.28:0/3295408706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf38004800 0x7fcf38004c10 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcf340018d0 tx=0x7fcf38005830 comp rx=0 tx=0).stop 2024-04-25T19:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.080+0000 7fcf4ad9d640 1 -- 172.21.15.28:0/3295408706 reap_dead start 2024-04-25T19:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.080+0000 7fcf50884640 1 -- 172.21.15.28:0/3295408706 shutdown_connections 2024-04-25T19:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.080+0000 7fcf50884640 1 -- 172.21.15.28:0/3295408706 >> 172.21.15.28:0/3295408706 conn(0x7fcf4c0d2c60 msgr2=0x7fcf4c0d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.080+0000 7fcf50884640 1 -- 172.21.15.28:0/3295408706 shutdown_connections 2024-04-25T19:11:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.080+0000 7fcf50884640 1 -- 172.21.15.28:0/3295408706 wait complete. 2024-04-25T19:11:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.082+0000 7fcf50884640 1 Processor -- start 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.082+0000 7fcf50884640 1 -- start start 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.083+0000 7fcf50884640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38004800 0x7fcf4c14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.083+0000 7fcf50884640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c163970 0x7fcf4c159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.083+0000 7fcf50884640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf4c159fa0 0x7fcf4c1f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.083+0000 7fcf50884640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c06e760 con 0x7fcf4c163970 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.083+0000 7fcf50884640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4c06e5e0 con 0x7fcf4c159fa0 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.083+0000 7fcf50884640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4c0b4d00 con 0x7fcf38004800 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c163970 0x7fcf4c159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c163970 0x7fcf4c159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56138/0 (socket says 172.21.15.28:56138) 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 -- 172.21.15.28:0/1610159581 learned_addr learned my addr 172.21.15.28:0/1610159581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf4ad9d640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf4c159fa0 0x7fcf4c1f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf4a59c640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38004800 0x7fcf4c14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 -- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38004800 msgr2=0x7fcf4c14df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38004800 0x7fcf4c14df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 -- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf4c159fa0 msgr2=0x7fcf4c1f61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf4c159fa0 0x7fcf4c1f61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 -- 172.21.15.28:0/1610159581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4c15a3b0 con 0x7fcf4c163970 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf4ad9d640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf4c159fa0 0x7fcf4c1f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf4a59c640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38004800 0x7fcf4c14df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf43fff640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c163970 0x7fcf4c159a60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcf340018d0 tx=0x7fcf340012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf42ffd640 1 -- 172.21.15.28:0/1610159581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3403f020 con 0x7fcf4c163970 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf42ffd640 1 -- 172.21.15.28:0/1610159581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf34006e50 con 0x7fcf4c163970 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf4c1615c0 con 0x7fcf4c163970 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf42ffd640 1 -- 172.21.15.28:0/1610159581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf34037040 con 0x7fcf4c163970 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.084+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf4c161b80 con 0x7fcf4c163970 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.085+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf4c06a100 con 0x7fcf4c163970 2024-04-25T19:11:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.086+0000 7fcf42ffd640 1 -- 172.21.15.28:0/1610159581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcf34042020 con 0x7fcf4c163970 2024-04-25T19:11:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.086+0000 7fcf42ffd640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcf2c0744b0 0x7fcf2c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.087+0000 7fcf4a59c640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcf2c0744b0 0x7fcf2c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.087+0000 7fcf42ffd640 1 -- 172.21.15.28:0/1610159581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fcf340b50a0 con 0x7fcf4c163970 2024-04-25T19:11:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.087+0000 7fcf4a59c640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcf2c0744b0 0x7fcf2c0769c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0d0650 tx=0x7fcf3c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.091+0000 7fcf42ffd640 1 -- 172.21.15.28:0/1610159581 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcf34049030 con 0x7fcf4c163970 2024-04-25T19:11:30.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.285+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fcf4c068e10 con 0x7fcf4c163970 2024-04-25T19:11:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.932+0000 7fcf42ffd640 1 -- 172.21.15.28:0/1610159581 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7fcf4c15a3b0 con 0x7fcf4c163970 2024-04-25T19:11:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcf2c0744b0 msgr2=0x7fcf2c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcf2c0744b0 0x7fcf2c0769c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0d0650 tx=0x7fcf3c009040 comp rx=0 tx=0).stop 2024-04-25T19:11:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c163970 msgr2=0x7fcf4c159a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c163970 0x7fcf4c159a60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcf340018d0 tx=0x7fcf340012d0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 shutdown_connections 2024-04-25T19:11:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcf2c0744b0 0x7fcf2c0769c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf4c159fa0 0x7fcf4c1f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf4c163970 0x7fcf4c159a60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 --2- 172.21.15.28:0/1610159581 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcf38004800 0x7fcf4c14df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 >> 172.21.15.28:0/1610159581 conn(0x7fcf4c0d2c60 msgr2=0x7fcf4c1601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 shutdown_connections 2024-04-25T19:11:30.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:30.935+0000 7fcf50884640 1 -- 172.21.15.28:0/1610159581 wait complete. 2024-04-25T19:11:30.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T19:11:31.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.076+0000 7fb1359c7640 1 Processor -- start 2024-04-25T19:11:31.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.076+0000 7fb1359c7640 1 -- start start 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb1359c7640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb1300b1d60 0x7fb1300b2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb1359c7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300b4410 0x7fb1300b4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb1359c7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1300b26b0 con 0x7fb1300d2840 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb1359c7640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fb13006d580 con 0x7fb1300b1d60 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb1359c7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb1300d1400 con 0x7fb1300b4410 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb12f7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb1300d2840 0x7fb1300d2c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33328/0 (socket says 172.21.15.28:33328) 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb12f7fe640 1 -- 172.21.15.28:0/622823973 learned_addr learned my addr 172.21.15.28:0/622823973 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb12effd640 1 --2- 172.21.15.28:0/622823973 >> v2:172.21.15.53:3300/0 conn(0x7fb1300b1d60 0x7fb1300b2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb12e7fc640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300b4410 0x7fb1300b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554806462 0 0) 0x7fb120001490 con 0x7fb1300d2840 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.077+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb118003520 con 0x7fb1300d2840 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3468113763 0 0) 0x7fb118003520 con 0x7fb1300d2840 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300b4410 msgr2=0x7fb1300b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300b4410 0x7fb1300b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 >> v2:172.21.15.53:3300/0 conn(0x7fb1300b1d60 msgr2=0x7fb1300b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 --2- 172.21.15.28:0/622823973 >> v2:172.21.15.53:3300/0 conn(0x7fb1300b1d60 0x7fb1300b2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1300d16e0 con 0x7fb1300d2840 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fb120001bc0 con 0x7fb1300d2840 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 >> v1:172.21.15.28:6789/0 conn(0x7fb1300d2840 legacy=0x7fb1300d2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300b4410 0x7fb1180039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118003f30 0x7fb118004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb118004d40 0x7fb118005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb13006d580 con 0x7fb118004d40 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb1300d1400 con 0x7fb118003f30 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb11800ba10 con 0x7fb1300b4410 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12effd640 1 --2- 172.21.15.28:0/622823973 >> v2:172.21.15.53:3300/0 conn(0x7fb1300b1d60 0x7fb1300b2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12f7fe640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb118004d40 0x7fb118005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12e7fc640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118003f30 0x7fb118004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12effd640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300b4410 0x7fb1180039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12f7fe640 1 -- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300b4410 msgr2=0x7fb1180039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12f7fe640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300b4410 0x7fb1180039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12f7fe640 1 -- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118003f30 msgr2=0x7fb118004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12f7fe640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb118003f30 0x7fb118004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.078+0000 7fb12f7fe640 1 -- 172.21.15.28:0/622823973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb118003520 con 0x7fb118004d40 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.079+0000 7fb12f7fe640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb118004d40 0x7fb118005150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb12000aa40 tx=0x7fb118005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab8e09e14659afc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.079+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12000a690 con 0x7fb118004d40 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.079+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb12002f070 con 0x7fb118004d40 2024-04-25T19:11:31.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.079+0000 7fb12dffb640 1 -- 172.21.15.28:0/622823973 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb120034470 con 0x7fb118004d40 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.080+0000 7fb1359c7640 1 -- 172.21.15.28:0/622823973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb118004d40 msgr2=0x7fb118005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.080+0000 7fb1359c7640 1 --2- 172.21.15.28:0/622823973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb118004d40 0x7fb118005150 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb12000aa40 tx=0x7fb118005830 comp rx=0 tx=0).stop 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.080+0000 7fb12f7fe640 1 -- 172.21.15.28:0/622823973 reap_dead start 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.080+0000 7fb1359c7640 1 -- 172.21.15.28:0/622823973 shutdown_connections 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.080+0000 7fb1359c7640 1 -- 172.21.15.28:0/622823973 >> 172.21.15.28:0/622823973 conn(0x7fb1300746c0 msgr2=0x7fb1300b76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.080+0000 7fb1359c7640 1 -- 172.21.15.28:0/622823973 shutdown_connections 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.080+0000 7fb1359c7640 1 -- 172.21.15.28:0/622823973 wait complete. 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.082+0000 7fb1359c7640 1 Processor -- start 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.082+0000 7fb1359c7640 1 -- start start 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb1359c7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1300cbbc0 0x7fb1300a0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb1359c7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300a0690 0x7fb13014cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb1359c7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1300cbfd0 0x7fb13014d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb1359c7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1300d1f80 con 0x7fb1300cbbc0 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb1359c7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb1300d1ca0 con 0x7fb1300cbfd0 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb1359c7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb1300d24a0 con 0x7fb1300a0690 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb12effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1300cbbc0 0x7fb1300a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb12effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1300cbbc0 0x7fb1300a0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36660/0 (socket says 172.21.15.28:36660) 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb12effd640 1 -- 172.21.15.28:0/2648711605 learned_addr learned my addr 172.21.15.28:0/2648711605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb12effd640 1 -- 172.21.15.28:0/2648711605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300a0690 msgr2=0x7fb13014cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb12effd640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300a0690 0x7fb13014cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb12effd640 1 -- 172.21.15.28:0/2648711605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1300cbfd0 msgr2=0x7fb13014d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb12effd640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1300cbfd0 0x7fb13014d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.083+0000 7fb12effd640 1 -- 172.21.15.28:0/2648711605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1300b03b0 con 0x7fb1300cbbc0 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.084+0000 7fb12effd640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1300cbbc0 0x7fb1300a0150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb128011cd0 tx=0x7fb12800f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.084+0000 7fb117fff640 1 -- 172.21.15.28:0/2648711605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb12800fca0 con 0x7fb1300cbbc0 2024-04-25T19:11:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.084+0000 7fb117fff640 1 -- 172.21.15.28:0/2648711605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb12800fe40 con 0x7fb1300cbbc0 2024-04-25T19:11:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.084+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1300b0690 con 0x7fb1300cbbc0 2024-04-25T19:11:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.084+0000 7fb117fff640 1 -- 172.21.15.28:0/2648711605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb128014680 con 0x7fb1300cbbc0 2024-04-25T19:11:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.084+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1300b0c50 con 0x7fb1300cbbc0 2024-04-25T19:11:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.085+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb130050740 con 0x7fb1300cbbc0 2024-04-25T19:11:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.086+0000 7fb117fff640 1 -- 172.21.15.28:0/2648711605 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb128012040 con 0x7fb1300cbbc0 2024-04-25T19:11:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.086+0000 7fb117fff640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb11c0744e0 0x7fb11c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.086+0000 7fb117fff640 1 -- 172.21.15.28:0/2648711605 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7fb128095b30 con 0x7fb1300cbbc0 2024-04-25T19:11:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.086+0000 7fb12e7fc640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb11c0744e0 0x7fb11c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:31.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.087+0000 7fb12e7fc640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb11c0744e0 0x7fb11c0769f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb13015da20 tx=0x7fb1240095d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.091+0000 7fb117fff640 1 -- 172.21.15.28:0/2648711605 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb128019030 con 0x7fb1300cbbc0 2024-04-25T19:11:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:31.274+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb13015d4d0 con 0x7fb1300cbbc0 2024-04-25T19:11:32.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:11:32.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:11:32.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.944+0000 7fb117fff640 1 -- 172.21.15.28:0/2648711605 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7fb128060220 con 0x7fb1300cbbc0 2024-04-25T19:11:32.947 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' created 2024-04-25T19:11:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb11c0744e0 msgr2=0x7fb11c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb11c0744e0 0x7fb11c0769f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb13015da20 tx=0x7fb1240095d0 comp rx=0 tx=0).stop 2024-04-25T19:11:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1300cbbc0 msgr2=0x7fb1300a0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1300cbbc0 0x7fb1300a0150 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb128011cd0 tx=0x7fb12800f610 comp rx=0 tx=0).stop 2024-04-25T19:11:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 shutdown_connections 2024-04-25T19:11:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb11c0744e0 0x7fb11c0769f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1300cbfd0 0x7fb13014d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1300a0690 0x7fb13014cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 --2- 172.21.15.28:0/2648711605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1300cbbc0 0x7fb1300a0150 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 >> 172.21.15.28:0/2648711605 conn(0x7fb1300746c0 msgr2=0x7fb1300b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 shutdown_connections 2024-04-25T19:11:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:32.948+0000 7fb1359c7640 1 -- 172.21.15.28:0/2648711605 wait complete. 2024-04-25T19:11:32.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T19:11:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.090+0000 7f2e4b461640 1 Processor -- start 2024-04-25T19:11:33.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.090+0000 7f2e4b461640 1 -- start start 2024-04-25T19:11:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.091+0000 7f2e4b461640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2e440b9210 0x7f2e4406e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.091+0000 7f2e4b461640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440d1e40 0x7f2e440d2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.091+0000 7f2e4b461640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e440cdd10 con 0x7f2e4406e9d0 2024-04-25T19:11:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.091+0000 7f2e4b461640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f2e440d37a0 con 0x7f2e440b9210 2024-04-25T19:11:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.091+0000 7f2e4b461640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2e4409cf80 con 0x7f2e440d1e40 2024-04-25T19:11:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e489fc640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2e4406e9d0 0x7f2e440ce980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33334/0 (socket says 172.21.15.28:33334) 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e489fc640 1 -- 172.21.15.28:0/651571668 learned_addr learned my addr 172.21.15.28:0/651571668 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e491fd640 1 --2- 172.21.15.28:0/651571668 >> v2:172.21.15.53:3300/0 conn(0x7f2e440b9210 0x7f2e4406e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e499fe640 1 --2- 172.21.15.28:0/651571668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440d1e40 0x7f2e440d2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e3bfff640 1 -- 172.21.15.28:0/651571668 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4156933318 0 0) 0x7f2e440cdd10 con 0x7f2e4406e9d0 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e3bfff640 1 -- 172.21.15.28:0/651571668 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e30003520 con 0x7f2e4406e9d0 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e3bfff640 1 -- 172.21.15.28:0/651571668 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2368530289 0 0) 0x7f2e30003520 con 0x7f2e4406e9d0 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e499fe640 1 -- 172.21.15.28:0/651571668 >> v2:172.21.15.53:3300/0 conn(0x7f2e440b9210 msgr2=0x7f2e4406e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e499fe640 1 --2- 172.21.15.28:0/651571668 >> v2:172.21.15.53:3300/0 conn(0x7f2e440b9210 0x7f2e4406e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e499fe640 1 -- 172.21.15.28:0/651571668 >> v1:172.21.15.28:6789/0 conn(0x7f2e4406e9d0 legacy=0x7f2e440ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e499fe640 1 -- 172.21.15.28:0/651571668 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4406de00 con 0x7f2e440d1e40 2024-04-25T19:11:33.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.092+0000 7f2e491fd640 1 --2- 172.21.15.28:0/651571668 >> v2:172.21.15.53:3300/0 conn(0x7f2e440b9210 0x7f2e4406e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:33.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.093+0000 7f2e499fe640 1 --2- 172.21.15.28:0/651571668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440d1e40 0x7f2e440d2250 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009d90 tx=0x7f2e3c02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=415b3e54154649eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:33.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.093+0000 7f2e3bfff640 1 -- 172.21.15.28:0/651571668 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34001670 con 0x7f2e440d1e40 2024-04-25T19:11:33.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.093+0000 7f2e3bfff640 1 -- 172.21.15.28:0/651571668 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e3c007540 con 0x7f2e440d1e40 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.093+0000 7f2e3bfff640 1 -- 172.21.15.28:0/651571668 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c0078f0 con 0x7f2e440d1e40 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.094+0000 7f2e4b461640 1 -- 172.21.15.28:0/651571668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440d1e40 msgr2=0x7f2e440d2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.094+0000 7f2e4b461640 1 --2- 172.21.15.28:0/651571668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440d1e40 0x7f2e440d2250 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009d90 tx=0x7f2e3c02a560 comp rx=0 tx=0).stop 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.094+0000 7f2e4b461640 1 -- 172.21.15.28:0/651571668 shutdown_connections 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.094+0000 7f2e4b461640 1 --2- 172.21.15.28:0/651571668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440d1e40 0x7f2e440d2250 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.094+0000 7f2e4b461640 1 --2- 172.21.15.28:0/651571668 >> v2:172.21.15.53:3300/0 conn(0x7f2e440b9210 0x7f2e4406e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.094+0000 7f2e4b461640 1 -- 172.21.15.28:0/651571668 >> 172.21.15.28:0/651571668 conn(0x7f2e440d03a0 msgr2=0x7f2e440d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.094+0000 7f2e4b461640 1 -- 172.21.15.28:0/651571668 shutdown_connections 2024-04-25T19:11:33.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.094+0000 7f2e4b461640 1 -- 172.21.15.28:0/651571668 wait complete. 2024-04-25T19:11:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.096+0000 7f2e4b461640 1 Processor -- start 2024-04-25T19:11:33.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.096+0000 7f2e4b461640 1 -- start start 2024-04-25T19:11:33.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e4b461640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e4406e9d0 0x7f2e441fa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:33.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e4b461640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440b9210 0x7f2e440cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:33.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e4b461640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440d1e40 0x7f2e4414c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:33.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e4b461640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2e440d1040 con 0x7f2e440d1e40 2024-04-25T19:11:33.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e4b461640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2e440d0ec0 con 0x7f2e4406e9d0 2024-04-25T19:11:33.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e4b461640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2e440cf090 con 0x7f2e440b9210 2024-04-25T19:11:33.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e499fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440d1e40 0x7f2e4414c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e499fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440d1e40 0x7f2e4414c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36668/0 (socket says 172.21.15.28:36668) 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e499fe640 1 -- 172.21.15.28:0/1189320925 learned_addr learned my addr 172.21.15.28:0/1189320925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e491fd640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e4406e9d0 0x7f2e441fa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.097+0000 7f2e489fc640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440b9210 0x7f2e440cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e499fe640 1 -- 172.21.15.28:0/1189320925 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440b9210 msgr2=0x7f2e440cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e499fe640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440b9210 0x7f2e440cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e499fe640 1 -- 172.21.15.28:0/1189320925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e4406e9d0 msgr2=0x7f2e441fa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e499fe640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e4406e9d0 0x7f2e441fa100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e499fe640 1 -- 172.21.15.28:0/1189320925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e440b6e00 con 0x7f2e440d1e40 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e499fe640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440d1e40 0x7f2e4414c820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009eb0 tx=0x7f2e3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e39ffb640 1 -- 172.21.15.28:0/1189320925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c03f070 con 0x7f2e440d1e40 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e39ffb640 1 -- 172.21.15.28:0/1189320925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e3c031c40 con 0x7f2e440d1e40 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e440b70e0 con 0x7f2e440d1e40 2024-04-25T19:11:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e39ffb640 1 -- 172.21.15.28:0/1189320925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c009450 con 0x7f2e440d1e40 2024-04-25T19:11:33.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.098+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e440b76a0 con 0x7f2e440d1e40 2024-04-25T19:11:33.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.099+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e4406a100 con 0x7f2e440d1e40 2024-04-25T19:11:33.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.100+0000 7f2e39ffb640 1 -- 172.21.15.28:0/1189320925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2e3c043410 con 0x7f2e440d1e40 2024-04-25T19:11:33.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.100+0000 7f2e39ffb640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2e280744b0 0x7f2e280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:33.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.101+0000 7f2e39ffb640 1 -- 172.21.15.28:0/1189320925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7f2e3c0b6810 con 0x7f2e440d1e40 2024-04-25T19:11:33.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.101+0000 7f2e491fd640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2e280744b0 0x7f2e280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:33.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.101+0000 7f2e491fd640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2e280744b0 0x7f2e280769c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009500 tx=0x7f2e40009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:33.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.106+0000 7f2e39ffb640 1 -- 172.21.15.28:0/1189320925 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e3c080590 con 0x7f2e440d1e40 2024-04-25T19:11:33.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.291+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f2e44072e80 con 0x7f2e440d1e40 2024-04-25T19:11:33.466 INFO:tasks.ceph.mds.d.smithi053.stderr:2024-04-25T19:11:33.464+0000 7fc58dcd3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T19:11:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.947+0000 7f2e39ffb640 1 -- 172.21.15.28:0/1189320925 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7f2e3c084890 con 0x7f2e440d1e40 2024-04-25T19:11:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T19:11:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2e280744b0 msgr2=0x7f2e280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2e280744b0 0x7f2e280769c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2e40009500 tx=0x7f2e40009110 comp rx=0 tx=0).stop 2024-04-25T19:11:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440d1e40 msgr2=0x7f2e4414c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440d1e40 0x7f2e4414c820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009eb0 tx=0x7f2e3c004a40 comp rx=0 tx=0).stop 2024-04-25T19:11:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 shutdown_connections 2024-04-25T19:11:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2e280744b0 0x7f2e280769c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440d1e40 0x7f2e4414c820 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2e440b9210 0x7f2e440cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 --2- 172.21.15.28:0/1189320925 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e4406e9d0 0x7f2e441fa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.954+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 >> 172.21.15.28:0/1189320925 conn(0x7f2e440d03a0 msgr2=0x7f2e440d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.955+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 shutdown_connections 2024-04-25T19:11:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:33.955+0000 7f2e4b461640 1 -- 172.21.15.28:0/1189320925 wait complete. 2024-04-25T19:11:33.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T19:11:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.102+0000 7fe33b5b2640 1 Processor -- start 2024-04-25T19:11:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.102+0000 7fe33b5b2640 1 -- start start 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.103+0000 7fe33b5b2640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe334169870 0x7fe33416bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.103+0000 7fe33b5b2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe33409d1a0 con 0x7fe3340b32d0 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.103+0000 7fe33b5b2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3340d4870 con 0x7fe33409cd90 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.103+0000 7fe33b5b2640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fe33406df50 con 0x7fe334169870 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.103+0000 7fe338b4d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe3340b32d0 0x7fe3340b36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33348/0 (socket says 172.21.15.28:33348) 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.103+0000 7fe338b4d640 1 -- 172.21.15.28:0/2342000136 learned_addr learned my addr 172.21.15.28:0/2342000136 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.103+0000 7fe339b4f640 1 --2- 172.21.15.28:0/2342000136 >> v2:172.21.15.53:3300/0 conn(0x7fe334169870 0x7fe33416bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2609512349 0 0) 0x7fe33409d1a0 con 0x7fe3340b32d0 2024-04-25T19:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe31c003520 con 0x7fe3340b32d0 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247661416 0 0) 0x7fe3340d4870 con 0x7fe33409cd90 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe33409d1a0 con 0x7fe33409cd90 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3409440770 0 0) 0x7fe31c003520 con 0x7fe3340b32d0 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 >> v2:172.21.15.53:3300/0 conn(0x7fe334169870 msgr2=0x7fe33416bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 --2- 172.21.15.28:0/2342000136 >> v2:172.21.15.53:3300/0 conn(0x7fe334169870 0x7fe33416bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 >> v1:172.21.15.53:6789/0 conn(0x7fe33409cd90 legacy=0x7fe33406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe33406d960 con 0x7fe3340b32d0 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fe328001670 con 0x7fe3340b32d0 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe339b4f640 1 --2- 172.21.15.28:0/2342000136 >> v2:172.21.15.53:3300/0 conn(0x7fe334169870 0x7fe33416bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 >> v1:172.21.15.28:6789/0 conn(0x7fe3340b32d0 legacy=0x7fe3340b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c003b40 0x7fe31c003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c004800 0x7fe31c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe31c005260 0x7fe31c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe33406df50 con 0x7fe31c003b40 2024-04-25T19:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c00c1c0 con 0x7fe31c005260 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe31c00c3c0 con 0x7fe31c004800 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe33934e640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c003b40 0x7fe31c003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe339b4f640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe31c005260 0x7fe31c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.104+0000 7fe338b4d640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c004800 0x7fe31c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe33934e640 1 -- 172.21.15.28:0/2342000136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c004800 msgr2=0x7fe31c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe33934e640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c004800 0x7fe31c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe33934e640 1 -- 172.21.15.28:0/2342000136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe31c005260 msgr2=0x7fe31c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe33934e640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe31c005260 0x7fe31c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe339b4f640 1 -- 172.21.15.28:0/2342000136 reap_dead start 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe33934e640 1 -- 172.21.15.28:0/2342000136 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c003520 con 0x7fe31c003b40 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe33934e640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c003b40 0x7fe31c003f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe32c001fd0 tx=0x7fe31c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f646100f8a7c61bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c001bf0 con 0x7fe31c003b40 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe32c003b70 con 0x7fe31c003b40 2024-04-25T19:11:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.105+0000 7fe333fff640 1 -- 172.21.15.28:0/2342000136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c0050d0 con 0x7fe31c003b40 2024-04-25T19:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.106+0000 7fe33b5b2640 1 -- 172.21.15.28:0/2342000136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c003b40 msgr2=0x7fe31c003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.106+0000 7fe33b5b2640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c003b40 0x7fe31c003f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe32c001fd0 tx=0x7fe31c005830 comp rx=0 tx=0).stop 2024-04-25T19:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.106+0000 7fe33b5b2640 1 -- 172.21.15.28:0/2342000136 shutdown_connections 2024-04-25T19:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.106+0000 7fe33b5b2640 1 --2- 172.21.15.28:0/2342000136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe31c003b40 0x7fe31c003f90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.106+0000 7fe33b5b2640 1 -- 172.21.15.28:0/2342000136 >> 172.21.15.28:0/2342000136 conn(0x7fe3340d03f0 msgr2=0x7fe3340d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.106+0000 7fe33b5b2640 1 -- 172.21.15.28:0/2342000136 shutdown_connections 2024-04-25T19:11:34.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.106+0000 7fe33b5b2640 1 -- 172.21.15.28:0/2342000136 wait complete. 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.108+0000 7fe33b5b2640 1 Processor -- start 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.108+0000 7fe33b5b2640 1 -- start start 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe33b5b2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c003b40 0x7fe33415dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe33b5b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3340c74a0 0x7fe3340b0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe33b5b2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3340b23f0 0x7fe3340b2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe33b5b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3340b9180 con 0x7fe3340c74a0 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe33b5b2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe33406dc70 con 0x7fe3340b23f0 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe33b5b2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe3340b9460 con 0x7fe31c003b40 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe338b4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3340c74a0 0x7fe3340b0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe338b4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3340c74a0 0x7fe3340b0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36678/0 (socket says 172.21.15.28:36678) 2024-04-25T19:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe338b4d640 1 -- 172.21.15.28:0/1602611086 learned_addr learned my addr 172.21.15.28:0/1602611086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.110+0000 7fe339b4f640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3340b23f0 0x7fe3340b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.111+0000 7fe338b4d640 1 -- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c003b40 msgr2=0x7fe33415dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.111+0000 7fe33934e640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c003b40 0x7fe33415dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.111+0000 7fe338b4d640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c003b40 0x7fe33415dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.111+0000 7fe338b4d640 1 -- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3340b23f0 msgr2=0x7fe3340b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.111+0000 7fe338b4d640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3340b23f0 0x7fe3340b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.111+0000 7fe338b4d640 1 -- 172.21.15.28:0/1602611086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3340b2d40 con 0x7fe3340c74a0 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.111+0000 7fe338b4d640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3340c74a0 0x7fe3340b0830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3280017e0 tx=0x7fe328007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.111+0000 7fe33934e640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c003b40 0x7fe33415dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.113+0000 7fe331ffb640 1 -- 172.21.15.28:0/1602611086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32800f7b0 con 0x7fe3340c74a0 2024-04-25T19:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.113+0000 7fe331ffb640 1 -- 172.21.15.28:0/1602611086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe328018070 con 0x7fe3340c74a0 2024-04-25T19:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.113+0000 7fe331ffb640 1 -- 172.21.15.28:0/1602611086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32800fbb0 con 0x7fe3340c74a0 2024-04-25T19:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.113+0000 7fe33b5b2640 1 -- 172.21.15.28:0/1602611086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe334161de0 con 0x7fe3340c74a0 2024-04-25T19:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.113+0000 7fe33b5b2640 1 -- 172.21.15.28:0/1602611086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3340d2ea0 con 0x7fe3340c74a0 2024-04-25T19:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.116+0000 7fe31b7fe640 1 -- 172.21.15.28:0/1602611086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe334050740 con 0x7fe3340c74a0 2024-04-25T19:11:34.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.116+0000 7fe331ffb640 1 -- 172.21.15.28:0/1602611086 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe32800fd50 con 0x7fe3340c74a0 2024-04-25T19:11:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.117+0000 7fe331ffb640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe320074500 0x7fe320076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.117+0000 7fe331ffb640 1 -- 172.21.15.28:0/1602611086 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7fe3280952e0 con 0x7fe3340c74a0 2024-04-25T19:11:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.117+0000 7fe33934e640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe320074500 0x7fe320076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:34.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.118+0000 7fe33934e640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe320074500 0x7fe320076a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe32c032000 tx=0x7fe32c02f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:34.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.122+0000 7fe331ffb640 1 -- 172.21.15.28:0/1602611086 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3340b2d40 con 0x7fe3340c74a0 2024-04-25T19:11:34.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.347+0000 7fe31b7fe640 1 -- 172.21.15.28:0/1602611086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fe3340b6a10 con 0x7fe3340c74a0 2024-04-25T19:11:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.963+0000 7fe331ffb640 1 -- 172.21.15.28:0/1602611086 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7fe32805fa10 con 0x7fe3340c74a0 2024-04-25T19:11:34.965 INFO:teuthology.orchestra.run.smithi028.stderr:added data pool 4 to fsmap 2024-04-25T19:11:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.965+0000 7fe31b7fe640 1 -- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe320074500 msgr2=0x7fe320076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:34.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.965+0000 7fe31b7fe640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe320074500 0x7fe320076a10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe32c032000 tx=0x7fe32c02f000 comp rx=0 tx=0).stop 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.965+0000 7fe31b7fe640 1 -- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3340c74a0 msgr2=0x7fe3340b0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.965+0000 7fe31b7fe640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3340c74a0 0x7fe3340b0830 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe3280017e0 tx=0x7fe328007430 comp rx=0 tx=0).stop 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.966+0000 7fe31b7fe640 1 -- 172.21.15.28:0/1602611086 shutdown_connections 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.966+0000 7fe31b7fe640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe320074500 0x7fe320076a10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.966+0000 7fe31b7fe640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3340b23f0 0x7fe3340b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.966+0000 7fe31b7fe640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3340c74a0 0x7fe3340b0830 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.966+0000 7fe31b7fe640 1 --2- 172.21.15.28:0/1602611086 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe31c003b40 0x7fe33415dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.966+0000 7fe31b7fe640 1 -- 172.21.15.28:0/1602611086 >> 172.21.15.28:0/1602611086 conn(0x7fe3340d03f0 msgr2=0x7fe334072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.966+0000 7fe31b7fe640 1 -- 172.21.15.28:0/1602611086 shutdown_connections 2024-04-25T19:11:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:34.966+0000 7fe31b7fe640 1 -- 172.21.15.28:0/1602611086 wait complete. 2024-04-25T19:11:34.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.097+0000 7fc9d9fa8640 1 Processor -- start 2024-04-25T19:11:35.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.097+0000 7fc9d9fa8640 1 -- start start 2024-04-25T19:11:35.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d9fa8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d40b4c80 0x7fc9d4160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d9fa8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc9d4164800 0x7fc9d40b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d9fa8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9d40d15b0 con 0x7fc9d40b78c0 2024-04-25T19:11:35.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d9fa8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fc9d4164c10 con 0x7fc9d4164800 2024-04-25T19:11:35.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d9fa8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d40d12d0 con 0x7fc9d40b4c80 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d2ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc9d4164800 0x7fc9d40b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d2ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc9d4164800 0x7fc9d40b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36692/0 (socket says 172.21.15.28:36692) 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d3fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc9d40b78c0 0x7fc9d40b7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33360/0 (socket says 172.21.15.28:33360) 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d2ffd640 1 -- 172.21.15.28:0/2634586339 learned_addr learned my addr 172.21.15.28:0/2634586339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d37fe640 1 --2- 172.21.15.28:0/2634586339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d40b4c80 0x7fc9d4160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d2ffd640 1 -- 172.21.15.28:0/2634586339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d40b4c80 msgr2=0x7fc9d4160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d2ffd640 1 --2- 172.21.15.28:0/2634586339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d40b4c80 0x7fc9d4160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d2ffd640 1 -- 172.21.15.28:0/2634586339 >> v1:172.21.15.28:6789/0 conn(0x7fc9d40b78c0 legacy=0x7fc9d40b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d2ffd640 1 -- 172.21.15.28:0/2634586339 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d4163d20 con 0x7fc9d4164800 2024-04-25T19:11:35.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.099+0000 7fc9d27fc640 1 -- 172.21.15.28:0/2634586339 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 159350649 0 0) 0x7fc9d40d15b0 con 0x7fc9d40b78c0 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.100+0000 7fc9d2ffd640 1 --2- 172.21.15.28:0/2634586339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d4164800 0x7fc9d40b46c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009850 tx=0x7fc9c802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f399231fc743a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.100+0000 7fc9d27fc640 1 -- 172.21.15.28:0/2634586339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8004030 con 0x7fc9d4164800 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.100+0000 7fc9d27fc640 1 -- 172.21.15.28:0/2634586339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c80041d0 con 0x7fc9d4164800 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.100+0000 7fc9d27fc640 1 -- 172.21.15.28:0/2634586339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8003610 con 0x7fc9d4164800 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.101+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/2634586339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d4164800 msgr2=0x7fc9d40b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.101+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/2634586339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d4164800 0x7fc9d40b46c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009850 tx=0x7fc9c802a230 comp rx=0 tx=0).stop 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.101+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/2634586339 shutdown_connections 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.101+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/2634586339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d4164800 0x7fc9d40b46c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.101+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/2634586339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d40b4c80 0x7fc9d4160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.101+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/2634586339 >> 172.21.15.28:0/2634586339 conn(0x7fc9d40d2980 msgr2=0x7fc9d40d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.101+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/2634586339 shutdown_connections 2024-04-25T19:11:35.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.101+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/2634586339 wait complete. 2024-04-25T19:11:35.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.103+0000 7fc9d9fa8640 1 Processor -- start 2024-04-25T19:11:35.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.103+0000 7fc9d9fa8640 1 -- start start 2024-04-25T19:11:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d9fa8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d40b4c80 0x7fc9d40ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d9fa8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9d40b78c0 0x7fc9d414cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d9fa8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d4164800 0x7fc9d415bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d9fa8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d40b5240 con 0x7fc9d40b4c80 2024-04-25T19:11:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d9fa8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d41642e0 con 0x7fc9d40b78c0 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d9fa8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc9d40b53c0 con 0x7fc9d4164800 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d40b4c80 0x7fc9d40ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d40b4c80 0x7fc9d40ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36702/0 (socket says 172.21.15.28:36702) 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.104+0000 7fc9d37fe640 1 -- 172.21.15.28:0/422298809 learned_addr learned my addr 172.21.15.28:0/422298809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d2ffd640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9d40b78c0 0x7fc9d414cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d3fff640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d4164800 0x7fc9d415bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d37fe640 1 -- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d4164800 msgr2=0x7fc9d415bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d37fe640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d4164800 0x7fc9d415bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d37fe640 1 -- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9d40b78c0 msgr2=0x7fc9d414cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d37fe640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9d40b78c0 0x7fc9d414cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d37fe640 1 -- 172.21.15.28:0/422298809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d4161b00 con 0x7fc9d40b4c80 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d2ffd640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9d40b78c0 0x7fc9d414cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:35.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d3fff640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d4164800 0x7fc9d415bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d37fe640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d40b4c80 0x7fc9d40ccb90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400e8a0 tx=0x7fc9c400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9bbfff640 1 -- 172.21.15.28:0/422298809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4017070 con 0x7fc9d40b4c80 2024-04-25T19:11:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9bbfff640 1 -- 172.21.15.28:0/422298809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9c400c940 con 0x7fc9d40b4c80 2024-04-25T19:11:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d4161de0 con 0x7fc9d40b4c80 2024-04-25T19:11:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9bbfff640 1 -- 172.21.15.28:0/422298809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c400cd30 con 0x7fc9d40b4c80 2024-04-25T19:11:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.105+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d41623a0 con 0x7fc9d40b4c80 2024-04-25T19:11:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.106+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9d4050740 con 0x7fc9d40b4c80 2024-04-25T19:11:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.107+0000 7fc9bbfff640 1 -- 172.21.15.28:0/422298809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc9c401a070 con 0x7fc9d40b4c80 2024-04-25T19:11:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.107+0000 7fc9bbfff640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc9c00744e0 0x7fc9c00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.107+0000 7fc9bbfff640 1 -- 172.21.15.28:0/422298809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc9c4095770 con 0x7fc9d40b4c80 2024-04-25T19:11:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.107+0000 7fc9d2ffd640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc9c00744e0 0x7fc9c00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.108+0000 7fc9d2ffd640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc9c00744e0 0x7fc9c00769f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8033000 tx=0x7fc9c802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.112+0000 7fc9bbfff640 1 -- 172.21.15.28:0/422298809 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc9c405fc50 con 0x7fc9d40b4c80 2024-04-25T19:11:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.332+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9d415fa60 con 0x7fc9d40b4c80 2024-04-25T19:11:35.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.333+0000 7fc9bbfff640 1 -- 172.21.15.28:0/422298809 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5174 (secure 0 0 0) 0x7fc9c405f550 con 0x7fc9d40b4c80 2024-04-25T19:11:35.335 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:35.335 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:34.348783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:11:35.335 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc9c00744e0 msgr2=0x7fc9c00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc9c00744e0 0x7fc9c00769f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8033000 tx=0x7fc9c802e000 comp rx=0 tx=0).stop 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d40b4c80 msgr2=0x7fc9d40ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d40b4c80 0x7fc9d40ccb90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400e8a0 tx=0x7fc9c400ed60 comp rx=0 tx=0).stop 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 shutdown_connections 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc9c00744e0 0x7fc9c00769f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc9d4164800 0x7fc9d415bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc9d40b78c0 0x7fc9d414cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 --2- 172.21.15.28:0/422298809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9d40b4c80 0x7fc9d40ccb90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 >> 172.21.15.28:0/422298809 conn(0x7fc9d40d2980 msgr2=0x7fc9d40d43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 shutdown_connections 2024-04-25T19:11:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.335+0000 7fc9d9fa8640 1 -- 172.21.15.28:0/422298809 wait complete. 2024-04-25T19:11:35.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:35.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.457+0000 7f22649c5640 1 Processor -- start 2024-04-25T19:11:35.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.457+0000 7f22649c5640 1 -- start start 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.458+0000 7f22649c5640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f226015aa00 0x7f226006dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.458+0000 7f22649c5640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22600b4110 0x7f22600b4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.458+0000 7f22649c5640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f226006f000 con 0x7f226006e4e0 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.458+0000 7f22649c5640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f226006ea40 con 0x7f226015aa00 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.458+0000 7f22649c5640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f226006ed20 con 0x7f22600b4110 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f226015aa00 0x7f226006dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f226015aa00 0x7f226006dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36704/0 (socket says 172.21.15.28:36704) 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225e59c640 1 -- 172.21.15.28:0/3028094364 learned_addr learned my addr 172.21.15.28:0/3028094364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225dd9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f226006e4e0 0x7f22600b0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33366/0 (socket says 172.21.15.28:33366) 2024-04-25T19:11:35.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225ed9d640 1 --2- 172.21.15.28:0/3028094364 >> v2:172.21.15.53:3300/0 conn(0x7f22600b4110 0x7f22600b4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225e59c640 1 -- 172.21.15.28:0/3028094364 >> v2:172.21.15.53:3300/0 conn(0x7f22600b4110 msgr2=0x7f22600b4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:35.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225e59c640 1 --2- 172.21.15.28:0/3028094364 >> v2:172.21.15.53:3300/0 conn(0x7f22600b4110 0x7f22600b4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225e59c640 1 -- 172.21.15.28:0/3028094364 >> v1:172.21.15.28:6789/0 conn(0x7f226006e4e0 legacy=0x7f22600b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225d59a640 1 -- 172.21.15.28:0/3028094364 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1757287168 0 0) 0x7f226006f000 con 0x7f226006e4e0 2024-04-25T19:11:35.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.459+0000 7f225e59c640 1 -- 172.21.15.28:0/3028094364 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22600b27e0 con 0x7f226015aa00 2024-04-25T19:11:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.460+0000 7f225e59c640 1 --2- 172.21.15.28:0/3028094364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f226015aa00 0x7f226006dfa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2260074180 tx=0x7f225402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15822307cfc8db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.460+0000 7f225d59a640 1 -- 172.21.15.28:0/3028094364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254037070 con 0x7f226015aa00 2024-04-25T19:11:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.460+0000 7f225d59a640 1 -- 172.21.15.28:0/3028094364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2254032440 con 0x7f226015aa00 2024-04-25T19:11:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.460+0000 7f225d59a640 1 -- 172.21.15.28:0/3028094364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2254032810 con 0x7f226015aa00 2024-04-25T19:11:35.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.460+0000 7f22649c5640 1 -- 172.21.15.28:0/3028094364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f226015aa00 msgr2=0x7f226006dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.460+0000 7f22649c5640 1 --2- 172.21.15.28:0/3028094364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f226015aa00 0x7f226006dfa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2260074180 tx=0x7f225402a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.461+0000 7f22649c5640 1 -- 172.21.15.28:0/3028094364 shutdown_connections 2024-04-25T19:11:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.461+0000 7f22649c5640 1 --2- 172.21.15.28:0/3028094364 >> v2:172.21.15.53:3300/0 conn(0x7f22600b4110 0x7f22600b4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.461+0000 7f22649c5640 1 --2- 172.21.15.28:0/3028094364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f226015aa00 0x7f226006dfa0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.461+0000 7f22649c5640 1 -- 172.21.15.28:0/3028094364 >> 172.21.15.28:0/3028094364 conn(0x7f22600b75d0 msgr2=0x7f22600b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.461+0000 7f22649c5640 1 -- 172.21.15.28:0/3028094364 shutdown_connections 2024-04-25T19:11:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.461+0000 7f22649c5640 1 -- 172.21.15.28:0/3028094364 wait complete. 2024-04-25T19:11:35.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.463+0000 7f22649c5640 1 Processor -- start 2024-04-25T19:11:35.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.463+0000 7f22649c5640 1 -- start start 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f22649c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f226006e4e0 0x7f226013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f22649c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22600b4110 0x7f2260154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f22649c5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f226015aa00 0x7f22601552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f22649c5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f226015ae10 con 0x7f22600b4110 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f22649c5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f226015d7d0 con 0x7f226015aa00 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f22649c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f22600b4a60 con 0x7f226006e4e0 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22600b4110 0x7f2260154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225dd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22600b4110 0x7f2260154d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36716/0 (socket says 172.21.15.28:36716) 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225dd9b640 1 -- 172.21.15.28:0/3274403484 learned_addr learned my addr 172.21.15.28:0/3274403484 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:35.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225ed9d640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f226015aa00 0x7f22601552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225e59c640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f226006e4e0 0x7f226013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225dd9b640 1 -- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f226006e4e0 msgr2=0x7f226013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225dd9b640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f226006e4e0 0x7f226013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225dd9b640 1 -- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f226015aa00 msgr2=0x7f22601552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225dd9b640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f226015aa00 0x7f22601552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225dd9b640 1 -- 172.21.15.28:0/3274403484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22601f9310 con 0x7f22600b4110 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.464+0000 7f225e59c640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f226006e4e0 0x7f226013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.465+0000 7f225dd9b640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22600b4110 0x7f2260154d80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f225000c420 tx=0x7f225000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.465+0000 7f224effd640 1 -- 172.21.15.28:0/3274403484 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22500074e0 con 0x7f22600b4110 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.465+0000 7f224effd640 1 -- 172.21.15.28:0/3274403484 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22500047e0 con 0x7f22600b4110 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.465+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22601f8550 con 0x7f22600b4110 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.465+0000 7f224effd640 1 -- 172.21.15.28:0/3274403484 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250004c10 con 0x7f22600b4110 2024-04-25T19:11:35.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.465+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22601f8b10 con 0x7f22600b4110 2024-04-25T19:11:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.466+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22601514c0 con 0x7f22600b4110 2024-04-25T19:11:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.467+0000 7f224effd640 1 -- 172.21.15.28:0/3274403484 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2250011070 con 0x7f22600b4110 2024-04-25T19:11:35.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.467+0000 7f224effd640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22400744b0 0x7f22400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.467+0000 7f224effd640 1 -- 172.21.15.28:0/3274403484 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f22500950a0 con 0x7f22600b4110 2024-04-25T19:11:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.467+0000 7f225e59c640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22400744b0 0x7f22400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.468+0000 7f225e59c640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22400744b0 0x7f22400769c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2254034000 tx=0x7f225402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.473+0000 7f224effd640 1 -- 172.21.15.28:0/3274403484 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f225005f580 con 0x7f22600b4110 2024-04-25T19:11:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.658+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f226006d830 con 0x7f22600b4110 2024-04-25T19:11:35.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.659+0000 7f224effd640 1 -- 172.21.15.28:0/3274403484 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f225005ee80 con 0x7f22600b4110 2024-04-25T19:11:35.661 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:35.661 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:35.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22400744b0 msgr2=0x7f22400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22400744b0 0x7f22400769c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2254034000 tx=0x7f225402e000 comp rx=0 tx=0).stop 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22600b4110 msgr2=0x7f2260154d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22600b4110 0x7f2260154d80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f225000c420 tx=0x7f225000c8e0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 shutdown_connections 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22400744b0 0x7f22400769c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f226015aa00 0x7f22601552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22600b4110 0x7f2260154d80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 --2- 172.21.15.28:0/3274403484 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f226006e4e0 0x7f226013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 >> 172.21.15.28:0/3274403484 conn(0x7f22600b75d0 msgr2=0x7f2260157520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 shutdown_connections 2024-04-25T19:11:35.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.661+0000 7f22649c5640 1 -- 172.21.15.28:0/3274403484 wait complete. 2024-04-25T19:11:35.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.784+0000 7f45f4a0c640 1 Processor -- start 2024-04-25T19:11:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.784+0000 7f45f4a0c640 1 -- start start 2024-04-25T19:11:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45f4a0c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45f00b7fd0 0x7f45f0160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45f4a0c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f45f0164800 0x7f45f00b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45f4a0c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f00d15b0 con 0x7f45f00b4c80 2024-04-25T19:11:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45f4a0c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f45f0164c10 con 0x7f45f00b7fd0 2024-04-25T19:11:35.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45f4a0c640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f45f00d12d0 con 0x7f45f0164800 2024-04-25T19:11:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45edd9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45f00b7fd0 0x7f45f0160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45edd9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45f00b7fd0 0x7f45f0160220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36732/0 (socket says 172.21.15.28:36732) 2024-04-25T19:11:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45edd9b640 1 -- 172.21.15.28:0/3825105679 learned_addr learned my addr 172.21.15.28:0/3825105679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45eed9d640 1 --2- 172.21.15.28:0/3825105679 >> v2:172.21.15.53:3300/0 conn(0x7f45f0164800 0x7f45f00b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45edd9b640 1 -- 172.21.15.28:0/3825105679 >> v2:172.21.15.53:3300/0 conn(0x7f45f0164800 msgr2=0x7f45f00b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45edd9b640 1 --2- 172.21.15.28:0/3825105679 >> v2:172.21.15.53:3300/0 conn(0x7f45f0164800 0x7f45f00b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45edd9b640 1 -- 172.21.15.28:0/3825105679 >> v1:172.21.15.53:6789/0 conn(0x7f45f00b4c80 legacy=0x7f45f00b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:35.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.786+0000 7f45edd9b640 1 -- 172.21.15.28:0/3825105679 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f0163d20 con 0x7f45f00b7fd0 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.787+0000 7f45edd9b640 1 --2- 172.21.15.28:0/3825105679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f00b7fd0 0x7f45f0160220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f45f00d56e0 tx=0x7f45d802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162f7dc8b7188f83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.787+0000 7f45ed59a640 1 -- 172.21.15.28:0/3825105679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8004030 con 0x7f45f00b7fd0 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.787+0000 7f45ed59a640 1 -- 172.21.15.28:0/3825105679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45d80041d0 con 0x7f45f00b7fd0 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.787+0000 7f45ed59a640 1 -- 172.21.15.28:0/3825105679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8003610 con 0x7f45f00b7fd0 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.788+0000 7f45f4a0c640 1 -- 172.21.15.28:0/3825105679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f00b7fd0 msgr2=0x7f45f0160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.788+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/3825105679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f00b7fd0 0x7f45f0160220 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f45f00d56e0 tx=0x7f45d802a210 comp rx=0 tx=0).stop 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.788+0000 7f45f4a0c640 1 -- 172.21.15.28:0/3825105679 shutdown_connections 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.788+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/3825105679 >> v2:172.21.15.53:3300/0 conn(0x7f45f0164800 0x7f45f00b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.788+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/3825105679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f00b7fd0 0x7f45f0160220 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.788+0000 7f45f4a0c640 1 -- 172.21.15.28:0/3825105679 >> 172.21.15.28:0/3825105679 conn(0x7f45f00d2980 msgr2=0x7f45f00d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:35.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.788+0000 7f45f4a0c640 1 -- 172.21.15.28:0/3825105679 shutdown_connections 2024-04-25T19:11:35.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.788+0000 7f45f4a0c640 1 -- 172.21.15.28:0/3825105679 wait complete. 2024-04-25T19:11:35.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.790+0000 7f45f4a0c640 1 Processor -- start 2024-04-25T19:11:35.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.790+0000 7f45f4a0c640 1 -- start start 2024-04-25T19:11:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45f4a0c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45f00b4c80 0x7f45f0162390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45f4a0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f0164800 0x7f45f015aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45f4a0c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45f0160780 0x7f45f0160b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45f4a0c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45f00b5240 con 0x7f45f0164800 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45f4a0c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f45f01642e0 con 0x7f45f0160780 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45f4a0c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f45f00b53c0 con 0x7f45f00b4c80 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45edd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f0164800 0x7f45f015aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45edd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f0164800 0x7f45f015aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36748/0 (socket says 172.21.15.28:36748) 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.791+0000 7f45edd9b640 1 -- 172.21.15.28:0/2026327947 learned_addr learned my addr 172.21.15.28:0/2026327947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45ee59c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45f00b4c80 0x7f45f0162390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45eed9d640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45f0160780 0x7f45f0160b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45edd9b640 1 -- 172.21.15.28:0/2026327947 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45f00b4c80 msgr2=0x7f45f0162390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45edd9b640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45f00b4c80 0x7f45f0162390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45edd9b640 1 -- 172.21.15.28:0/2026327947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45f0160780 msgr2=0x7f45f0160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45edd9b640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45f0160780 0x7f45f0160b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45edd9b640 1 -- 172.21.15.28:0/2026327947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f01628d0 con 0x7f45f0164800 2024-04-25T19:11:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45edd9b640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f0164800 0x7f45f015aeb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009940 tx=0x7f45d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45e6ffd640 1 -- 172.21.15.28:0/2026327947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8041030 con 0x7f45f0164800 2024-04-25T19:11:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45e6ffd640 1 -- 172.21.15.28:0/2026327947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45d8003c50 con 0x7f45f0164800 2024-04-25T19:11:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f0161600 con 0x7f45f0164800 2024-04-25T19:11:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45e6ffd640 1 -- 172.21.15.28:0/2026327947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8004030 con 0x7f45f0164800 2024-04-25T19:11:35.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.792+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f00cba70 con 0x7f45f0164800 2024-04-25T19:11:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.793+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f0050740 con 0x7f45f0164800 2024-04-25T19:11:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.794+0000 7f45e6ffd640 1 -- 172.21.15.28:0/2026327947 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f45d8044070 con 0x7f45f0164800 2024-04-25T19:11:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.794+0000 7f45e6ffd640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f45d00744b0 0x7f45d00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.794+0000 7f45e6ffd640 1 -- 172.21.15.28:0/2026327947 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f45d80b60d0 con 0x7f45f0164800 2024-04-25T19:11:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.794+0000 7f45ee59c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f45d00744b0 0x7f45d00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:35.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.795+0000 7f45ee59c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f45d00744b0 0x7f45d00769c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009f20 tx=0x7f45e00094d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.799+0000 7f45e6ffd640 1 -- 172.21.15.28:0/2026327947 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45d80805b0 con 0x7f45f0164800 2024-04-25T19:11:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.988+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45f00d3f70 con 0x7f45f0164800 2024-04-25T19:11:35.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.989+0000 7f45e6ffd640 1 -- 172.21.15.28:0/2026327947 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f45d807feb0 con 0x7f45f0164800 2024-04-25T19:11:35.991 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:35.991 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f45d00744b0 msgr2=0x7f45d00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f45d00744b0 0x7f45d00769c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009f20 tx=0x7f45e00094d0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f0164800 msgr2=0x7f45f015aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f0164800 0x7f45f015aeb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f45d8009940 tx=0x7f45d80046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 shutdown_connections 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f45d00744b0 0x7f45d00769c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45f0160780 0x7f45f0160b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f0164800 0x7f45f015aeb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 --2- 172.21.15.28:0/2026327947 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f45f00b4c80 0x7f45f0162390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 >> 172.21.15.28:0/2026327947 conn(0x7f45f00d2980 msgr2=0x7f45f00d4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:35.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 shutdown_connections 2024-04-25T19:11:35.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:35.991+0000 7f45f4a0c640 1 -- 172.21.15.28:0/2026327947 wait complete. 2024-04-25T19:11:36.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:11:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.128+0000 7f24fdca8640 1 Processor -- start 2024-04-25T19:11:36.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.128+0000 7f24fdca8640 1 -- start start 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.129+0000 7f24fdca8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24f806edb0 0x7f24f80b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.129+0000 7f24fdca8640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f24f80d1b30 0x7f24f80d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.129+0000 7f24fdca8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f8160c00 con 0x7f24f81616a0 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.129+0000 7f24fdca8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f24f809c190 con 0x7f24f806edb0 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.129+0000 7f24fdca8640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f24f809bc50 con 0x7f24f80d1b30 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f77fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24f806edb0 0x7f24f80b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f77fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24f806edb0 0x7f24f80b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36758/0 (socket says 172.21.15.28:36758) 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f77fe640 1 -- 172.21.15.28:0/87150569 learned_addr learned my addr 172.21.15.28:0/87150569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f77fe640 1 -- 172.21.15.28:0/87150569 >> v2:172.21.15.53:3300/0 conn(0x7f24f80d1b30 msgr2=0x7f24f80d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f7fff640 1 --2- 172.21.15.28:0/87150569 >> v2:172.21.15.53:3300/0 conn(0x7f24f80d1b30 0x7f24f80d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f77fe640 1 --2- 172.21.15.28:0/87150569 >> v2:172.21.15.53:3300/0 conn(0x7f24f80d1b30 0x7f24f80d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f77fe640 1 -- 172.21.15.28:0/87150569 >> v1:172.21.15.53:6789/0 conn(0x7f24f81616a0 legacy=0x7f24f809b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f77fe640 1 -- 172.21.15.28:0/87150569 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f809beb0 con 0x7f24f806edb0 2024-04-25T19:11:36.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.130+0000 7f24f7fff640 1 --2- 172.21.15.28:0/87150569 >> v2:172.21.15.53:3300/0 conn(0x7f24f80d1b30 0x7f24f80d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:36.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.131+0000 7f24f77fe640 1 --2- 172.21.15.28:0/87150569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f80b3c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f24f80d13c0 tx=0x7f24e802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4a0f370931bb67d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:36.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.131+0000 7f24f67fc640 1 -- 172.21.15.28:0/87150569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8037070 con 0x7f24f806edb0 2024-04-25T19:11:36.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.131+0000 7f24f67fc640 1 -- 172.21.15.28:0/87150569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e8032440 con 0x7f24f806edb0 2024-04-25T19:11:36.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.131+0000 7f24f67fc640 1 -- 172.21.15.28:0/87150569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8032810 con 0x7f24f806edb0 2024-04-25T19:11:36.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.131+0000 7f24fdca8640 1 -- 172.21.15.28:0/87150569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 msgr2=0x7f24f80b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.131+0000 7f24fdca8640 1 --2- 172.21.15.28:0/87150569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f80b3c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f24f80d13c0 tx=0x7f24e802a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.132+0000 7f24fdca8640 1 -- 172.21.15.28:0/87150569 shutdown_connections 2024-04-25T19:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.132+0000 7f24fdca8640 1 --2- 172.21.15.28:0/87150569 >> v2:172.21.15.53:3300/0 conn(0x7f24f80d1b30 0x7f24f80d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.132+0000 7f24fdca8640 1 --2- 172.21.15.28:0/87150569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f80b3c40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.132+0000 7f24fdca8640 1 -- 172.21.15.28:0/87150569 >> 172.21.15.28:0/87150569 conn(0x7f24f80b7620 msgr2=0x7f24f80b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.132+0000 7f24fdca8640 1 -- 172.21.15.28:0/87150569 shutdown_connections 2024-04-25T19:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.132+0000 7f24fdca8640 1 -- 172.21.15.28:0/87150569 wait complete. 2024-04-25T19:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.133+0000 7f24fdca8640 1 Processor -- start 2024-04-25T19:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.134+0000 7f24fdca8640 1 -- start start 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24fdca8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f814f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24fdca8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24f80d1b30 0x7f24f8162590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24fdca8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24f81616a0 0x7f24f8162ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24fdca8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24f80b4180 con 0x7f24f806edb0 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24fdca8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f24f8160f00 con 0x7f24f81616a0 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24fdca8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f24f80d2480 con 0x7f24f80d1b30 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f814f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f814f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36766/0 (socket says 172.21.15.28:36766) 2024-04-25T19:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f77fe640 1 -- 172.21.15.28:0/553008897 learned_addr learned my addr 172.21.15.28:0/553008897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f7fff640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24f81616a0 0x7f24f8162ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f77fe640 1 -- 172.21.15.28:0/553008897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24f80d1b30 msgr2=0x7f24f8162590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f77fe640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24f80d1b30 0x7f24f8162590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f77fe640 1 -- 172.21.15.28:0/553008897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24f81616a0 msgr2=0x7f24f8162ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f77fe640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24f81616a0 0x7f24f8162ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.135+0000 7f24f77fe640 1 -- 172.21.15.28:0/553008897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f81fb2a0 con 0x7f24f806edb0 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.136+0000 7f24f77fe640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f814f640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24e80097f0 tx=0x7f24e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.136+0000 7f24dffff640 1 -- 172.21.15.28:0/553008897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8037040 con 0x7f24f806edb0 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.136+0000 7f24dffff640 1 -- 172.21.15.28:0/553008897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e8004230 con 0x7f24f806edb0 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.136+0000 7f24dffff640 1 -- 172.21.15.28:0/553008897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e803c730 con 0x7f24f806edb0 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.136+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f814fb80 con 0x7f24f806edb0 2024-04-25T19:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.136+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f81583c0 con 0x7f24f806edb0 2024-04-25T19:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.137+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f8050740 con 0x7f24f806edb0 2024-04-25T19:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.137+0000 7f24dffff640 1 -- 172.21.15.28:0/553008897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f24e803a070 con 0x7f24f806edb0 2024-04-25T19:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.138+0000 7f24dffff640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24e40744e0 0x7f24e40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.138+0000 7f24dffff640 1 -- 172.21.15.28:0/553008897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f24e80b5d60 con 0x7f24f806edb0 2024-04-25T19:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.138+0000 7f24f6ffd640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24e40744e0 0x7f24e40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.138+0000 7f24f6ffd640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24e40744e0 0x7f24e40769f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f24ec006fd0 tx=0x7f24ec006bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.142+0000 7f24dffff640 1 -- 172.21.15.28:0/553008897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f24e8041020 con 0x7f24f806edb0 2024-04-25T19:11:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.364+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24f80d2260 con 0x7f24f806edb0 2024-04-25T19:11:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.365+0000 7f24dffff640 1 -- 172.21.15.28:0/553008897 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5174 (secure 0 0 0) 0x7f24e80802d0 con 0x7f24f806edb0 2024-04-25T19:11:36.367 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:36.367 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:34.348783+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:11:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-04-25T19:11:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24e40744e0 msgr2=0x7f24e40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24e40744e0 0x7f24e40769f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f24ec006fd0 tx=0x7f24ec006bc0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 msgr2=0x7f24f814f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f814f640 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f24e80097f0 tx=0x7f24e80046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 shutdown_connections 2024-04-25T19:11:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24e40744e0 0x7f24e40769f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24f81616a0 0x7f24f8162ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24f80d1b30 0x7f24f8162590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 --2- 172.21.15.28:0/553008897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24f806edb0 0x7f24f814f640 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 >> 172.21.15.28:0/553008897 conn(0x7f24f80b7620 msgr2=0x7f24f80b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 shutdown_connections 2024-04-25T19:11:36.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.367+0000 7f24fdca8640 1 -- 172.21.15.28:0/553008897 wait complete. 2024-04-25T19:11:36.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:36.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.489+0000 7f2985269640 1 Processor -- start 2024-04-25T19:11:36.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.489+0000 7f2985269640 1 -- start start 2024-04-25T19:11:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f2985269640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29800b7d60 0x7f29800b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f2985269640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2980164d60 con 0x7f29800b4e30 2024-04-25T19:11:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f2985269640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29800d0220 con 0x7f29800b9470 2024-04-25T19:11:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f2985269640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f29800d0b50 con 0x7f29800b7d60 2024-04-25T19:11:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f297f7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29800b7d60 0x7f29800b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f297f7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29800b7d60 0x7f29800b9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36772/0 (socket says 172.21.15.28:36772) 2024-04-25T19:11:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f297effd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29800b4e30 0x7f29800bb650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33378/0 (socket says 172.21.15.28:33378) 2024-04-25T19:11:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f297f7fe640 1 -- 172.21.15.28:0/3330522545 learned_addr learned my addr 172.21.15.28:0/3330522545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f297f7fe640 1 -- 172.21.15.28:0/3330522545 >> v1:172.21.15.53:6789/0 conn(0x7f29800b9470 legacy=0x7f29800b7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f297f7fe640 1 -- 172.21.15.28:0/3330522545 >> v1:172.21.15.28:6789/0 conn(0x7f29800b4e30 legacy=0x7f29800bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.491+0000 7f297f7fe640 1 -- 172.21.15.28:0/3330522545 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29800d0500 con 0x7f29800b7d60 2024-04-25T19:11:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.492+0000 7f297f7fe640 1 --2- 172.21.15.28:0/3330522545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b7d60 0x7f29800b9880 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2970009e50 tx=0x7f297002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef4edb0a86f5e1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.492+0000 7f297dffb640 1 -- 172.21.15.28:0/3330522545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2970036070 con 0x7f29800b7d60 2024-04-25T19:11:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.492+0000 7f297dffb640 1 -- 172.21.15.28:0/3330522545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2970031600 con 0x7f29800b7d60 2024-04-25T19:11:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.492+0000 7f297dffb640 1 -- 172.21.15.28:0/3330522545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29700319d0 con 0x7f29800b7d60 2024-04-25T19:11:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.493+0000 7f2985269640 1 -- 172.21.15.28:0/3330522545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b7d60 msgr2=0x7f29800b9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.493+0000 7f2985269640 1 --2- 172.21.15.28:0/3330522545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b7d60 0x7f29800b9880 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2970009e50 tx=0x7f297002a6d0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.493+0000 7f2985269640 1 -- 172.21.15.28:0/3330522545 shutdown_connections 2024-04-25T19:11:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.493+0000 7f2985269640 1 --2- 172.21.15.28:0/3330522545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b7d60 0x7f29800b9880 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.493+0000 7f2985269640 1 -- 172.21.15.28:0/3330522545 >> 172.21.15.28:0/3330522545 conn(0x7f29800d2650 msgr2=0x7f29800d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:36.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.493+0000 7f2985269640 1 -- 172.21.15.28:0/3330522545 shutdown_connections 2024-04-25T19:11:36.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.493+0000 7f2985269640 1 -- 172.21.15.28:0/3330522545 wait complete. 2024-04-25T19:11:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.495+0000 7f2985269640 1 Processor -- start 2024-04-25T19:11:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.495+0000 7f2985269640 1 -- start start 2024-04-25T19:11:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f2985269640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b4e30 0x7f298015f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f2985269640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29800b9470 0x7f298015cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f2985269640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29800cd850 0x7f29800cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f2985269640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29801649c0 con 0x7f29800b4e30 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f2985269640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f298008e3a0 con 0x7f29800cd850 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f2985269640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2980164b40 con 0x7f29800b9470 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b4e30 0x7f298015f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b4e30 0x7f298015f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36786/0 (socket says 172.21.15.28:36786) 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297effd640 1 -- 172.21.15.28:0/3061734191 learned_addr learned my addr 172.21.15.28:0/3061734191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297f7fe640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29800cd850 0x7f29800cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297e7fc640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29800b9470 0x7f298015cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297effd640 1 -- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29800b9470 msgr2=0x7f298015cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297effd640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29800b9470 0x7f298015cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297effd640 1 -- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29800cd850 msgr2=0x7f29800cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297effd640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29800cd850 0x7f29800cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297effd640 1 -- 172.21.15.28:0/3061734191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298015e050 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.496+0000 7f297e7fc640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29800b9470 0x7f298015cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.497+0000 7f297effd640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b4e30 0x7f298015f5f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f297800c470 tx=0x7f297800c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.497+0000 7f29777fe640 1 -- 172.21.15.28:0/3061734191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29780074e0 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.497+0000 7f29777fe640 1 -- 172.21.15.28:0/3061734191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2978007680 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.497+0000 7f29777fe640 1 -- 172.21.15.28:0/3061734191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978004a00 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.497+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f298015e550 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.497+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f298015eb10 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.498+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2980050740 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.499+0000 7f29777fe640 1 -- 172.21.15.28:0/3061734191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f297801a020 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.499+0000 7f29777fe640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f29600744b0 0x7f29600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.499+0000 7f29777fe640 1 -- 172.21.15.28:0/3061734191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2978094c20 con 0x7f29800b4e30 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.499+0000 7f297e7fc640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f29600744b0 0x7f29600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.500+0000 7f297e7fc640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f29600744b0 0x7f29600769c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2968006fd0 tx=0x7f2968008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:36.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.504+0000 7f29777fe640 1 -- 172.21.15.28:0/3061734191 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f297805f080 con 0x7f29800b4e30 2024-04-25T19:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.689+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f29800b8ab0 con 0x7f29800b4e30 2024-04-25T19:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.690+0000 7f29777fe640 1 -- 172.21.15.28:0/3061734191 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f2978067070 con 0x7f29800b4e30 2024-04-25T19:11:36.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:36.692 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f29600744b0 msgr2=0x7f29600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f29600744b0 0x7f29600769c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2968006fd0 tx=0x7f2968008040 comp rx=0 tx=0).stop 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b4e30 msgr2=0x7f298015f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b4e30 0x7f298015f5f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f297800c470 tx=0x7f297800c930 comp rx=0 tx=0).stop 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 shutdown_connections 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f29600744b0 0x7f29600769c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29800cd850 0x7f29800cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f29800b9470 0x7f298015cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 --2- 172.21.15.28:0/3061734191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29800b4e30 0x7f298015f5f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 >> 172.21.15.28:0/3061734191 conn(0x7f29800d2650 msgr2=0x7f29800b52e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 shutdown_connections 2024-04-25T19:11:36.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:36.692+0000 7f2985269640 1 -- 172.21.15.28:0/3061734191 wait complete. 2024-04-25T19:11:37.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T19:11:29.309189+0000', 'modified': '2024-04-25T19:11:34.348783+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4281': {'gid': 4281, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6835/3585928833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 3585928833}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 3585928833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4281, 'qdb_cluster': [4281]} 2024-04-25T19:11:37.108 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T19:11:37.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T19:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.224+0000 7fe979a96640 1 Processor -- start 2024-04-25T19:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.224+0000 7fe979a96640 1 -- start start 2024-04-25T19:11:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe979a96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160120 0x7fe9740b7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe979a96640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe9740b7e80 0x7fe974160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe979a96640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe974160530 con 0x7fe974160cc0 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe979a96640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fe974164580 con 0x7fe9740b7e80 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe979a96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe9740d5460 con 0x7fe974160120 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe972ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe9740b7e80 0x7fe974160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe972ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe9740b7e80 0x7fe974160780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36794/0 (socket says 172.21.15.28:36794) 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe972ffd640 1 -- 172.21.15.28:0/996550572 learned_addr learned my addr 172.21.15.28:0/996550572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe973fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe974160cc0 0x7fe9740d0800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33392/0 (socket says 172.21.15.28:33392) 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe9737fe640 1 --2- 172.21.15.28:0/996550572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160120 0x7fe9740b7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe972ffd640 1 -- 172.21.15.28:0/996550572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160120 msgr2=0x7fe9740b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe972ffd640 1 --2- 172.21.15.28:0/996550572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160120 0x7fe9740b7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe972ffd640 1 -- 172.21.15.28:0/996550572 >> v1:172.21.15.28:6789/0 conn(0x7fe974160cc0 legacy=0x7fe9740d0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe972ffd640 1 -- 172.21.15.28:0/996550572 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9740b8290 con 0x7fe9740b7e80 2024-04-25T19:11:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.226+0000 7fe9727fc640 1 -- 172.21.15.28:0/996550572 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2861156522 0 0) 0x7fe96c001490 con 0x7fe974160cc0 2024-04-25T19:11:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.227+0000 7fe972ffd640 1 --2- 172.21.15.28:0/996550572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9740b7e80 0x7fe974160780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe968009870 tx=0x7fe96802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a55e605d189a28b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.227+0000 7fe9727fc640 1 -- 172.21.15.28:0/996550572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968037070 con 0x7fe9740b7e80 2024-04-25T19:11:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.227+0000 7fe9727fc640 1 -- 172.21.15.28:0/996550572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe968032440 con 0x7fe9740b7e80 2024-04-25T19:11:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.227+0000 7fe9727fc640 1 -- 172.21.15.28:0/996550572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968032810 con 0x7fe9740b7e80 2024-04-25T19:11:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.228+0000 7fe979a96640 1 -- 172.21.15.28:0/996550572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9740b7e80 msgr2=0x7fe974160780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.228+0000 7fe979a96640 1 --2- 172.21.15.28:0/996550572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9740b7e80 0x7fe974160780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe968009870 tx=0x7fe96802a270 comp rx=0 tx=0).stop 2024-04-25T19:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.228+0000 7fe979a96640 1 -- 172.21.15.28:0/996550572 shutdown_connections 2024-04-25T19:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.228+0000 7fe979a96640 1 --2- 172.21.15.28:0/996550572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9740b7e80 0x7fe974160780 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.228+0000 7fe979a96640 1 --2- 172.21.15.28:0/996550572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160120 0x7fe9740b7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.228+0000 7fe979a96640 1 -- 172.21.15.28:0/996550572 >> 172.21.15.28:0/996550572 conn(0x7fe9740d2980 msgr2=0x7fe9740d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.228+0000 7fe979a96640 1 -- 172.21.15.28:0/996550572 shutdown_connections 2024-04-25T19:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.228+0000 7fe979a96640 1 -- 172.21.15.28:0/996550572 wait complete. 2024-04-25T19:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.229+0000 7fe979a96640 1 Processor -- start 2024-04-25T19:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.230+0000 7fe979a96640 1 -- start start 2024-04-25T19:11:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe979a96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9740b7e80 0x7fe9741fa530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe979a96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974160120 0x7fe9741ebf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe979a96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160cc0 0x7fe9741ec460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe979a96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe974163f30 con 0x7fe974160120 2024-04-25T19:11:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe979a96640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe974163c50 con 0x7fe9740b7e80 2024-04-25T19:11:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe979a96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe974164210 con 0x7fe974160cc0 2024-04-25T19:11:37.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974160120 0x7fe9741ebf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974160120 0x7fe9741ebf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36800/0 (socket says 172.21.15.28:36800) 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe972ffd640 1 -- 172.21.15.28:0/1134932258 learned_addr learned my addr 172.21.15.28:0/1134932258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe973fff640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160cc0 0x7fe9741ec460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe9737fe640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9740b7e80 0x7fe9741fa530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe972ffd640 1 -- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160cc0 msgr2=0x7fe9741ec460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe972ffd640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160cc0 0x7fe9741ec460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe972ffd640 1 -- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9740b7e80 msgr2=0x7fe9741fa530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe972ffd640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9740b7e80 0x7fe9741fa530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe972ffd640 1 -- 172.21.15.28:0/1134932258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9741faa70 con 0x7fe974160120 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.231+0000 7fe9737fe640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9740b7e80 0x7fe9741fa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.232+0000 7fe972ffd640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974160120 0x7fe9741ebf20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe9680090b0 tx=0x7fe9680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.232+0000 7fe95bfff640 1 -- 172.21.15.28:0/1134932258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968037040 con 0x7fe974160120 2024-04-25T19:11:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.232+0000 7fe95bfff640 1 -- 172.21.15.28:0/1134932258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe968004030 con 0x7fe974160120 2024-04-25T19:11:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.232+0000 7fe979a96640 1 -- 172.21.15.28:0/1134932258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97415c030 con 0x7fe974160120 2024-04-25T19:11:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.232+0000 7fe95bfff640 1 -- 172.21.15.28:0/1134932258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe968004400 con 0x7fe974160120 2024-04-25T19:11:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.232+0000 7fe979a96640 1 -- 172.21.15.28:0/1134932258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97415c2d0 con 0x7fe974160120 2024-04-25T19:11:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.234+0000 7fe95bfff640 1 -- 172.21.15.28:0/1134932258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe96803a070 con 0x7fe974160120 2024-04-25T19:11:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.234+0000 7fe95bfff640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe9600744e0 0x7fe9600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.234+0000 7fe95bfff640 1 -- 172.21.15.28:0/1134932258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe9680b5a30 con 0x7fe974160120 2024-04-25T19:11:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.234+0000 7fe9737fe640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe9600744e0 0x7fe9600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.235+0000 7fe9737fe640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe9600744e0 0x7fe9600769f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe9740d24c0 tx=0x7fe964008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.235+0000 7fe980a23740 1 Processor -- start 2024-04-25T19:11:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.235+0000 7fe980a23740 1 -- start start 2024-04-25T19:11:37.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe980a23740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d30919270 0x556d30ce5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe980a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f03b0 0x556d30ce1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe980a23740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 0x556d30cbb260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe980a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d30c1f740 con 0x556d308f03b0 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe980a23740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556d30bbb3a0 con 0x556d30919270 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe980a23740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x556d30c1b700 con 0x556d308f8590 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f03b0 0x556d30ce1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f03b0 0x556d30ce1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36806/0 (socket says 172.21.15.28:36806) 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 -- 172.21.15.28:0/2398120273 learned_addr learned my addr 172.21.15.28:0/2398120273 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe9737fe640 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d30919270 0x556d30ce5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe973fff640 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 0x556d30cbb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 -- 172.21.15.28:0/2398120273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 msgr2=0x556d30cbb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 0x556d30cbb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 -- 172.21.15.28:0/2398120273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d30919270 msgr2=0x556d30ce5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d30919270 0x556d30ce5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 -- 172.21.15.28:0/2398120273 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d30bd7750 con 0x556d308f03b0 2024-04-25T19:11:37.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.237+0000 7fe972ffd640 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f03b0 0x556d30ce1fa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe96807b940 tx=0x7fe96807b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20423d3d38b3f32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.238+0000 7fe9597fa640 1 -- 172.21.15.28:0/2398120273 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96804a040 con 0x556d308f03b0 2024-04-25T19:11:37.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.238+0000 7fe9597fa640 1 -- 172.21.15.28:0/2398120273 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe968032440 con 0x556d308f03b0 2024-04-25T19:11:37.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.238+0000 7fe9597fa640 1 -- 172.21.15.28:0/2398120273 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96807f3d0 con 0x556d308f03b0 2024-04-25T19:11:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.238+0000 7fe980a23740 1 -- 172.21.15.28:0/2398120273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f03b0 msgr2=0x556d30ce1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.238+0000 7fe980a23740 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f03b0 0x556d30ce1fa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe96807b940 tx=0x7fe96807b840 comp rx=0 tx=0).stop 2024-04-25T19:11:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.239+0000 7fe980a23740 1 -- 172.21.15.28:0/2398120273 shutdown_connections 2024-04-25T19:11:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.239+0000 7fe980a23740 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 0x556d30cbb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.239+0000 7fe980a23740 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f03b0 0x556d30ce1fa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.239+0000 7fe980a23740 1 --2- 172.21.15.28:0/2398120273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d30919270 0x556d30ce5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.239+0000 7fe980a23740 1 -- 172.21.15.28:0/2398120273 >> 172.21.15.28:0/2398120273 conn(0x556d3090d3c0 msgr2=0x556d30cba9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:37.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.239+0000 7fe980a23740 1 -- 172.21.15.28:0/2398120273 shutdown_connections 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.239+0000 7fe980a23740 1 -- 172.21.15.28:0/2398120273 wait complete. 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.239+0000 7fe980a23740 1 -- start start 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.240+0000 7fe980a23740 1 Processor -- start 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe980a23740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30ce8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe980a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f8590 0x556d30cdae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe980a23740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d30919270 0x556d30cdb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe980a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d30c1f740 con 0x556d308f8590 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe980a23740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556d30ba6fe0 con 0x556d308f03b0 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe980a23740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x556d30cfe8a0 con 0x556d30919270 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe980a23740 10 client.0 _ll_register_callbacks cb 0x556d30bbd150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f8590 0x556d30cdae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f8590 0x556d30cdae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36812/0 (socket says 172.21.15.28:36812) 2024-04-25T19:11:37.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 -- 172.21.15.28:0/447806626 learned_addr learned my addr 172.21.15.28:0/447806626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe9737fe640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30ce8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe973fff640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d30919270 0x556d30cdb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 -- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d30919270 msgr2=0x556d30cdb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d30919270 0x556d30cdb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 -- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 msgr2=0x556d30ce8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30ce8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 -- 172.21.15.28:0/447806626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d309312d0 con 0x556d308f8590 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe9737fe640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30ce8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe973fff640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d30919270 0x556d30cdb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe972ffd640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f8590 0x556d30cdae80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe96807f5c0 tx=0x7fe968009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad4f287b9aa107d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.241+0000 7fe9527fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe9527fc640 1 -- 172.21.15.28:0/447806626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96804a040 con 0x556d308f8590 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 1 -- 172.21.15.28:0/447806626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe968032440 con 0x556d308f8590 2024-04-25T19:11:37.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe9527fc640 1 -- 172.21.15.28:0/447806626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe968088150 con 0x556d308f8590 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe9527fc640 1 -- 172.21.15.28:0/447806626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fe96809e050 con 0x556d308f8590 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 10 client.4428 fetch_fsmap learned FSMap version 7 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 1 -- 172.21.15.28:0/447806626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x556d309fb6b0 con 0x556d308f8590 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe9527fc640 1 -- 172.21.15.28:0/447806626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe968032440 con 0x556d308f8590 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe9527fc640 1 -- 172.21.15.28:0/447806626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fe968080b40 con 0x556d308f8590 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 10 client.4428 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 10 client.4428 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 20 client.4428 populate_metadata read hostname 'smithi028' 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x556d308d7e40 0x556d30cdee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 4 client.4428 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe980a23740 1 -- 172.21.15.28:0/447806626 --> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d30c2d350 con 0x556d308d7e40 2024-04-25T19:11:37.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.242+0000 7fe9737fe640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x556d308d7e40 0x556d30cdee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.243+0000 7fe9527fc640 10 client.4428 ms_handle_connect on v2:172.21.15.53:6834/3585928833 2024-04-25T19:11:37.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.243+0000 7fe9737fe640 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x556d308d7e40 0x556d30cdee10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc4fbfa43e2267ff server_cookie=9fa77cfcdb620771 in_seq=0 out_seq=0 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe9527fc640 1 -- 172.21.15.28:0/447806626 <== mds.0 v2:172.21.15.53:6834/3585928833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556d30c2d350 con 0x556d308d7e40 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe9527fc640 10 client.4428 handle_command_reply: tid=0 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 client.4428 shutdown 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 -- 172.21.15.28:0/447806626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f8590 msgr2=0x556d30cdae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f8590 0x556d30cdae80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe96807f5c0 tx=0x7fe968009240 comp rx=0 tx=0).stop 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 -- 172.21.15.28:0/447806626 shutdown_connections 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x556d308d7e40 0x556d30cdee10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d30919270 0x556d30cdb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d308f8590 0x556d30cdae80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 --2- 172.21.15.28:0/447806626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30ce8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 -- 172.21.15.28:0/447806626 >> 172.21.15.28:0/447806626 conn(0x556d3090d3c0 msgr2=0x556d30cba9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:37.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe9527fc640 0 client.4428 ms_handle_reset on v2:172.21.15.53:6834/3585928833 2024-04-25T19:11:37.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 -- 172.21.15.28:0/447806626 shutdown_connections 2024-04-25T19:11:37.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 1 -- 172.21.15.28:0/447806626 wait complete. 2024-04-25T19:11:37.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.244+0000 7fe980a23740 20 client.4428 trim_cache size 0 max 16384 2024-04-25T19:11:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.263+0000 7fe980a23740 1 Processor -- start 2024-04-25T19:11:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.263+0000 7fe980a23740 1 -- start start 2024-04-25T19:11:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe980a23740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f03b0 0x556d30cb9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe980a23740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f8590 0x556d30cd8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe980a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cd9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe980a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe974203280 con 0x556d30919270 2024-04-25T19:11:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe980a23740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe9742027c0 con 0x556d308f8590 2024-04-25T19:11:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe980a23740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe974202c60 con 0x556d308f03b0 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cd9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cd9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36820/0 (socket says 172.21.15.28:36820) 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 -- 172.21.15.28:0/877440557 learned_addr learned my addr 172.21.15.28:0/877440557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe9737fe640 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f03b0 0x556d30cb9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe972ffd640 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f8590 0x556d30cd8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 -- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f03b0 msgr2=0x556d30cb9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f03b0 0x556d30cb9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 -- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f8590 msgr2=0x556d30cd8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f8590 0x556d30cd8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 -- 172.21.15.28:0/877440557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d30c21be0 con 0x556d30919270 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe9737fe640 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f03b0 0x556d30cb9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe972ffd640 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f8590 0x556d30cd8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe973fff640 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cd9000 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe9740d00e0 tx=0x7fe96c004c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad793cf52f5aebb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.265+0000 7fe958ff9640 1 -- 172.21.15.28:0/877440557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c0077b0 con 0x556d30919270 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe958ff9640 1 -- 172.21.15.28:0/877440557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96c01b020 con 0x556d30919270 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe958ff9640 1 -- 172.21.15.28:0/877440557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c0079b0 con 0x556d30919270 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 -- 172.21.15.28:0/877440557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 msgr2=0x556d30cd9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cd9000 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe9740d00e0 tx=0x7fe96c004c70 comp rx=0 tx=0).stop 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 -- 172.21.15.28:0/877440557 shutdown_connections 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cd9000 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f8590 0x556d30cd8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 --2- 172.21.15.28:0/877440557 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f03b0 0x556d30cb9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 -- 172.21.15.28:0/877440557 >> 172.21.15.28:0/877440557 conn(0x556d3090d3c0 msgr2=0x556d30d115f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 -- 172.21.15.28:0/877440557 shutdown_connections 2024-04-25T19:11:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.266+0000 7fe980a23740 1 -- 172.21.15.28:0/877440557 wait complete. 2024-04-25T19:11:37.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.267+0000 7fe980a23740 1 -- start start 2024-04-25T19:11:37.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.267+0000 7fe980a23740 1 Processor -- start 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.268+0000 7fe980a23740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30d21a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.268+0000 7fe980a23740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 0x556d30d21fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.268+0000 7fe980a23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cda6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe980a23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556d30cd9ec0 con 0x556d30919270 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe980a23740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x556d30c1f740 con 0x556d308f03b0 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe980a23740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x556d30cda520 con 0x556d308f8590 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe980a23740 10 client.0 _ll_register_callbacks cb 0x7fe964009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cda6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cda6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36832/0 (socket says 172.21.15.28:36832) 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 -- 172.21.15.28:0/2197668399 learned_addr learned my addr 172.21.15.28:0/2197668399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe972ffd640 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 0x556d30d21fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe9737fe640 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30d21a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 -- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 msgr2=0x556d30d21fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 0x556d30d21fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 -- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 msgr2=0x556d30d21a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30d21a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 -- 172.21.15.28:0/2197668399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d30cfcd10 con 0x556d30919270 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe973fff640 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cda6a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe96c019040 tx=0x7fe96c0015c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5055b2f093582405 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe9527fc640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe9527fc640 1 -- 172.21.15.28:0/2197668399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c01a030 con 0x556d30919270 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe9527fc640 1 -- 172.21.15.28:0/2197668399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe96c02adf0 con 0x556d30919270 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe980a23740 1 -- 172.21.15.28:0/2197668399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe96c01b020 con 0x556d30919270 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe9527fc640 1 -- 172.21.15.28:0/2197668399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7fe96c004db0 con 0x556d30919270 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe980a23740 10 client.4434 fetch_fsmap learned FSMap version 7 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.269+0000 7fe980a23740 1 -- 172.21.15.28:0/2197668399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x556d30d7ee70 con 0x556d30919270 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe9527fc640 1 -- 172.21.15.28:0/2197668399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96c01b020 con 0x556d30919270 2024-04-25T19:11:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe9527fc640 1 -- 172.21.15.28:0/2197668399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7fe96c0169e0 con 0x556d30919270 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe980a23740 10 client.4434 fetch_fsmap finished waiting for FSMap version 7 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe980a23740 10 client.4434 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe980a23740 20 client.4434 populate_metadata read hostname 'smithi028' 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe980a23740 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x556d30d7dd90 0x556d30d7e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe980a23740 4 client.4434 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe980a23740 1 -- 172.21.15.28:0/2197668399 --> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] -- command(tid 0: {"prefix": "status"}) -- 0x556d30ce7050 con 0x556d30d7dd90 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe9737fe640 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x556d30d7dd90 0x556d30d7e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe9737fe640 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x556d30d7dd90 0x556d30d7e1a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=758b8d9263caa4aa server_cookie=a0f382352207749b in_seq=0 out_seq=0 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.270+0000 7fe9527fc640 10 client.4434 ms_handle_connect on v2:172.21.15.53:6834/3585928833 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe9527fc640 1 -- 172.21.15.28:0/2197668399 <== mds.0 v2:172.21.15.53:6834/3585928833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x556d30ce7050 con 0x556d30d7dd90 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe9527fc640 10 client.4434 handle_command_reply: tid=0 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 client.4434 shutdown 2024-04-25T19:11:37.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 -- 172.21.15.28:0/2197668399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 msgr2=0x556d30cda6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cda6a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe96c019040 tx=0x7fe96c0015c0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 -- 172.21.15.28:0/2197668399 shutdown_connections 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x556d30d7dd90 0x556d30d7e1a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556d30919270 0x556d30cda6a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x556d308f8590 0x556d30d21fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 --2- 172.21.15.28:0/2197668399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x556d308f03b0 0x556d30d21a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 -- 172.21.15.28:0/2197668399 >> 172.21.15.28:0/2197668399 conn(0x556d3090d3c0 msgr2=0x556d30cd8ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe9527fc640 0 client.4434 ms_handle_reset on v2:172.21.15.53:6834/3585928833 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 -- 172.21.15.28:0/2197668399 shutdown_connections 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 1 -- 172.21.15.28:0/2197668399 wait complete. 2024-04-25T19:11:37.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe980a23740 20 client.4434 trim_cache size 0 max 16384 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "5d8007d4-24a5-41be-aa4d-cd64fedac62a", 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4281, 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 7.3562180289999999, 2024-04-25T19:11:37.280 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 7, 2024-04-25T19:11:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 24, 2024-04-25T19:11:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 24, 2024-04-25T19:11:37.281 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 14.799936701 2024-04-25T19:11:37.281 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:11:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe979a96640 1 -- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe9600744e0 msgr2=0x7fe9600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe979a96640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe9600744e0 0x7fe9600769f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe9740d24c0 tx=0x7fe964008040 comp rx=0 tx=0).stop 2024-04-25T19:11:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe979a96640 1 -- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974160120 msgr2=0x7fe9741ebf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.271+0000 7fe979a96640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974160120 0x7fe9741ebf20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe9680090b0 tx=0x7fe9680046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.272+0000 7fe979a96640 1 -- 172.21.15.28:0/1134932258 shutdown_connections 2024-04-25T19:11:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.272+0000 7fe979a96640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe9600744e0 0x7fe9600769f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.272+0000 7fe979a96640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe974160cc0 0x7fe9741ec460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.272+0000 7fe979a96640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe974160120 0x7fe9741ebf20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.272+0000 7fe979a96640 1 --2- 172.21.15.28:0/1134932258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9740b7e80 0x7fe9741fa530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.272+0000 7fe979a96640 1 -- 172.21.15.28:0/1134932258 >> 172.21.15.28:0/1134932258 conn(0x7fe9740d2980 msgr2=0x7fe9740b6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.272+0000 7fe979a96640 1 -- 172.21.15.28:0/1134932258 shutdown_connections 2024-04-25T19:11:37.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.272+0000 7fe979a96640 1 -- 172.21.15.28:0/1134932258 wait complete. 2024-04-25T19:11:37.285 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T19:11:37.285 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T19:11:37.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T19:11:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.395+0000 7f3b74e42640 1 Processor -- start 2024-04-25T19:11:37.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.395+0000 7f3b74e42640 1 -- start start 2024-04-25T19:11:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.396+0000 7f3b74e42640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b7009caf0 0x7f3b700d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.396+0000 7f3b74e42640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b700d1ce0 con 0x7f3b7009cf00 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.396+0000 7f3b74e42640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b700cf8d0 con 0x7f3b700d2670 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.396+0000 7f3b74e42640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f3b700b8fa0 con 0x7f3b7009caf0 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.396+0000 7f3b6e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b7009caf0 0x7f3b700d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.396+0000 7f3b6ed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3b7009cf00 0x7f3b700ce720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33404/0 (socket says 172.21.15.28:33404) 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.396+0000 7f3b6ed9d640 1 -- 172.21.15.28:0/1882544549 learned_addr learned my addr 172.21.15.28:0/1882544549 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.397+0000 7f3b6e59c640 1 -- 172.21.15.28:0/1882544549 >> v1:172.21.15.53:6789/0 conn(0x7f3b700d2670 legacy=0x7f3b700b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.397+0000 7f3b6e59c640 1 -- 172.21.15.28:0/1882544549 >> v1:172.21.15.28:6789/0 conn(0x7f3b7009cf00 legacy=0x7f3b700ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.397+0000 7f3b6e59c640 1 -- 172.21.15.28:0/1882544549 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b700cfbb0 con 0x7f3b7009caf0 2024-04-25T19:11:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.397+0000 7f3b6d59a640 1 -- 172.21.15.28:0/1882544549 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3029106305 0 0) 0x7f3b700d1ce0 con 0x7f3b7009cf00 2024-04-25T19:11:37.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.397+0000 7f3b6e59c640 1 --2- 172.21.15.28:0/1882544549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7009caf0 0x7f3b700d2130 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b600097a0 tx=0x7f3b6002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e46e14485c38c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.397+0000 7f3b6d59a640 1 -- 172.21.15.28:0/1882544549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60037070 con 0x7f3b7009caf0 2024-04-25T19:11:37.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.397+0000 7f3b6d59a640 1 -- 172.21.15.28:0/1882544549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b6002ae50 con 0x7f3b7009caf0 2024-04-25T19:11:37.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.398+0000 7f3b6d59a640 1 -- 172.21.15.28:0/1882544549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b60032660 con 0x7f3b7009caf0 2024-04-25T19:11:37.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.398+0000 7f3b74e42640 1 -- 172.21.15.28:0/1882544549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7009caf0 msgr2=0x7f3b700d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.398+0000 7f3b74e42640 1 --2- 172.21.15.28:0/1882544549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7009caf0 0x7f3b700d2130 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b600097a0 tx=0x7f3b6002a140 comp rx=0 tx=0).stop 2024-04-25T19:11:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.398+0000 7f3b74e42640 1 -- 172.21.15.28:0/1882544549 shutdown_connections 2024-04-25T19:11:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.398+0000 7f3b74e42640 1 --2- 172.21.15.28:0/1882544549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b7009caf0 0x7f3b700d2130 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.398+0000 7f3b74e42640 1 -- 172.21.15.28:0/1882544549 >> 172.21.15.28:0/1882544549 conn(0x7f3b700d03f0 msgr2=0x7f3b700d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.398+0000 7f3b74e42640 1 -- 172.21.15.28:0/1882544549 shutdown_connections 2024-04-25T19:11:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.398+0000 7f3b74e42640 1 -- 172.21.15.28:0/1882544549 wait complete. 2024-04-25T19:11:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.400+0000 7f3b74e42640 1 Processor -- start 2024-04-25T19:11:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.400+0000 7f3b74e42640 1 -- start start 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.401+0000 7f3b74e42640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b7009caf0 0x7f3b701dcf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b74e42640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b7009cf00 0x7f3b700b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b74e42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b700d2670 0x7f3b700a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b74e42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3b700b9430 con 0x7f3b700d2670 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b74e42640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3b700b78e0 con 0x7f3b7009cf00 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b74e42640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f3b700d4790 con 0x7f3b7009caf0 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b700d2670 0x7f3b700a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b700d2670 0x7f3b700a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36842/0 (socket says 172.21.15.28:36842) 2024-04-25T19:11:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 -- 172.21.15.28:0/2490154788 learned_addr learned my addr 172.21.15.28:0/2490154788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6dd9b640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b7009cf00 0x7f3b700b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6e59c640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b7009caf0 0x7f3b701dcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 -- 172.21.15.28:0/2490154788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b7009caf0 msgr2=0x7f3b701dcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b7009caf0 0x7f3b701dcf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 -- 172.21.15.28:0/2490154788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b7009cf00 msgr2=0x7f3b700b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b7009cf00 0x7f3b700b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 -- 172.21.15.28:0/2490154788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b700b3bf0 con 0x7f3b700d2670 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b6ed9d640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b700d2670 0x7f3b700a0820 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800caf0 tx=0x7f3b680021c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b5effd640 1 -- 172.21.15.28:0/2490154788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6800ac00 con 0x7f3b700d2670 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b5effd640 1 -- 172.21.15.28:0/2490154788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b6800ada0 con 0x7f3b700d2670 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b700b3ed0 con 0x7f3b700d2670 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b5effd640 1 -- 172.21.15.28:0/2490154788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b68012640 con 0x7f3b700d2670 2024-04-25T19:11:37.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.402+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b701dd4d0 con 0x7f3b700d2670 2024-04-25T19:11:37.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.403+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b700b3960 con 0x7f3b700d2670 2024-04-25T19:11:37.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.404+0000 7f3b5effd640 1 -- 172.21.15.28:0/2490154788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3b680127e0 con 0x7f3b700d2670 2024-04-25T19:11:37.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.404+0000 7f3b5effd640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3b500744e0 0x7f3b500769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.405+0000 7f3b6e59c640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3b500744e0 0x7f3b500769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.405+0000 7f3b5effd640 1 -- 172.21.15.28:0/2490154788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3b68095990 con 0x7f3b700d2670 2024-04-25T19:11:37.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.405+0000 7f3b6e59c640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3b500744e0 0x7f3b500769f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b60034000 tx=0x7f3b6002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.409+0000 7f3b5effd640 1 -- 172.21.15.28:0/2490154788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b6805f810 con 0x7f3b700d2670 2024-04-25T19:11:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.632+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3b70050740 con 0x7f3b700d2670 2024-04-25T19:11:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.632+0000 7f3b5effd640 1 -- 172.21.15.28:0/2490154788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f3b68063b10 con 0x7f3b700d2670 2024-04-25T19:11:37.635 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T19:11:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3b500744e0 msgr2=0x7f3b500769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3b500744e0 0x7f3b500769f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3b60034000 tx=0x7f3b6002e000 comp rx=0 tx=0).stop 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b700d2670 msgr2=0x7f3b700a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b700d2670 0x7f3b700a0820 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3b6800caf0 tx=0x7f3b680021c0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 shutdown_connections 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3b500744e0 0x7f3b500769f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b700d2670 0x7f3b700a0820 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b7009cf00 0x7f3b700b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 --2- 172.21.15.28:0/2490154788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3b7009caf0 0x7f3b701dcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.634+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 >> 172.21.15.28:0/2490154788 conn(0x7f3b700d03f0 msgr2=0x7f3b70071e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.635+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 shutdown_connections 2024-04-25T19:11:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37.635+0000 7f3b74e42640 1 -- 172.21.15.28:0/2490154788 wait complete. 2024-04-25T19:11:37.647 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T19:11:37.647 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T19:11:37.648 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-25T19:11:37.648 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T19:11:37.648 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T19:11:37.648 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T19:11:37.648 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T19:11:37.648 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T19:11:37.648 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T19:11:37.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:37.648 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-25T19:11:37.663 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-25T19:11:37.664 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-25T19:11:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T19:11:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42762sec preferred_lft 42762sec 2024-04-25T19:11:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-25T19:11:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:11:37.665 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:11:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-25T19:11:37.665 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-25T19:11:37.666 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T19:11:37.666 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:11:37.666 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:11:37.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-25T19:11:37.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-25T19:11:37.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-25T19:11:37.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T19:11:37.666 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:11:37.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:11:37.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:11:37.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:37.851 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T19:11:37.875 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-25T19:11:37.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:37.877 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-25T19:11:37.941 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-25T19:11:37.941 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T19:11:37.941 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T19:11:37.941 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T19:11:37.941 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T19:11:37.942 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:11:37.942 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:11:37.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T19:11:37.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T19:11:37.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T19:11:37.942 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:11:38.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:11:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:11:38.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:38.208 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:11:38.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:38.223 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-25T19:11:38.279 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:11:38.279 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:11:38.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T19:11:38.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T19:11:38.280 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:11:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:11:38.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:11:38.425 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T19:11:38.425 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:11:38.425 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:11:38.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T19:11:38.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T19:11:38.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T19:11:38.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T19:11:38.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T19:11:38.425 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:11:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:11:38.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:11:38.555 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:11:38.555 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:11:38.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-25T19:11:38.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T19:11:38.555 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:11:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:11:38.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:11:38.633 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T19:11:38.633 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:11:38.633 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T19:11:38.648 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T19:11:38.648 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:11:38.648 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T19:11:38.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-04-25T19:11:38.769 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T19:11:38.825 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-04-25T19:11:38.825 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-04-25T19:11:38.825 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-04-25T19:11:38.825 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-04-25T19:11:38.825 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-04-25T19:11:38.825 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-04-25T19:11:38.826 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T19:11:38.827 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T19:11:38.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:38.828 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-04-25T19:11:38.880 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T19:11:38.880 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T19:11:38.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo modprobe fuse 2024-04-25T19:11:38.947 DEBUG:teuthology.orchestra.run.smithi028:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/proc 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/sys 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/dev 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/security 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/shm 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/pts 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/run 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/cgroup 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/pstore 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/bpf 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/ 2024-04-25T19:11:39.003 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/selinux 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/mqueue 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/dev/hugepages 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/debug 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/tracing 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/kernel/config 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/sys/fs/fuse/connections 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/run/user/1000 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph 2024-04-25T19:11:39.004 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T19:11:39.005 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T19:11:39.005 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T19:11:39.005 INFO:teuthology.orchestra.run.smithi028.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T19:11:39.005 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns 2024-04-25T19:11:39.005 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T19:11:39.005 INFO:teuthology.orchestra.run.smithi028.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T19:11:39.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:39.006 DEBUG:teuthology.orchestra.run.smithi028:> ls /sys/fs/fuse/connections 2024-04-25T19:11:39.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:2024-04-25T19:11:39.020+0000 7f56751ca480 -1 init, newargv = 0x5577c3d75a80 newargc=15 2024-04-25T19:11:39.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[41467]: starting ceph client 2024-04-25T19:11:39.030 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[41467]: starting fuse 2024-04-25T19:11:39.037 INFO:teuthology.orchestra.run.smithi028.stdout:42 2024-04-25T19:11:39.038 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-25T19:11:39.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo stdin-killer -- python3 -c ' 2024-04-25T19:11:39.038 DEBUG:teuthology.orchestra.run.smithi028:> import glob 2024-04-25T19:11:39.038 DEBUG:teuthology.orchestra.run.smithi028:> import re 2024-04-25T19:11:39.038 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-04-25T19:11:39.038 DEBUG:teuthology.orchestra.run.smithi028:> import subprocess 2024-04-25T19:11:39.038 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T19:11:39.038 DEBUG:teuthology.orchestra.run.smithi028:> def _find_admin_socket(client_name): 2024-04-25T19:11:39.038 DEBUG:teuthology.orchestra.run.smithi028:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> files = glob.glob(asok_path) 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> # Given a non-glob path, it better be there 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> if "*" not in asok_path: 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> assert(len(files) == 1) 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> return files[0] 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> for f in files: 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> contents = proc_f.read() 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> if mountpoint in contents: 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> return f 2024-04-25T19:11:39.039 DEBUG:teuthology.orchestra.run.smithi028:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T19:11:39.040 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T19:11:39.040 DEBUG:teuthology.orchestra.run.smithi028:> print(_find_admin_socket("client.admin")) 2024-04-25T19:11:39.040 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-04-25T19:11:39.156 INFO:teuthology.orchestra.run.smithi028.stdout:/var/run/ceph/ceph-client.admin.41467.asok 2024-04-25T19:11:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:11:39.167 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.41467.asok 2024-04-25T19:11:39.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:39.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.41467.asok status 2024-04-25T19:11:39.313 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:11:39.313 INFO:teuthology.orchestra.run.smithi028.stdout: "metadata": { 2024-04-25T19:11:39.313 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T19:11:39.313 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T19:11:39.313 INFO:teuthology.orchestra.run.smithi028.stdout: "entity_id": "admin", 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": "smithi028", 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "pid": "41467", 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "root": "/" 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_count": 0, 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "dentry_pinned_count": 0, 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4449, 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "inst": { 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": { 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "client", 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "num": 4449 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-04-25T19:11:39.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4046673142 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": { 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "192.168.0.1:0", 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4046673142 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "inst_str": "client.4449 192.168.0.1:0/4046673142", 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "addr_str": "192.168.0.1:0/4046673142", 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "inode_count": 1, 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_epoch": 7, 2024-04-25T19:11:39.315 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch": 24, 2024-04-25T19:11:39.316 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_epoch_barrier": 0, 2024-04-25T19:11:39.316 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklisted": false, 2024-04-25T19:11:39.316 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs" 2024-04-25T19:11:39.316 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:11:39.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:39.322 DEBUG:teuthology.orchestra.run.smithi028:> stat --file-system '--printf=%T 2024-04-25T19:11:39.322 DEBUG:teuthology.orchestra.run.smithi028:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T19:11:39.337 INFO:teuthology.orchestra.run.smithi028.stdout:fuseblk 2024-04-25T19:11:39.338 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T19:11:39.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:39.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T19:11:39.404 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T19:11:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:11:39.520 INFO:teuthology.orchestra.run.smithi028.stdout:# file: . 2024-04-25T19:11:39.520 INFO:teuthology.orchestra.run.smithi028.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T19:11:39.520 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:39.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:11:39.527 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:39.527 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:11:39.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.693+0000 7fca9c3f3640 1 Processor -- start 2024-04-25T19:11:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.693+0000 7fca9c3f3640 1 -- start start 2024-04-25T19:11:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9c3f3640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca940b5130 0x7fca94161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9c3f3640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca9415f7d0 0x7fca9415fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9c3f3640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca94163e80 con 0x7fca9406ded0 2024-04-25T19:11:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9c3f3640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fca94163c30 con 0x7fca9415f7d0 2024-04-25T19:11:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9c3f3640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fca94164a90 con 0x7fca940b5130 2024-04-25T19:11:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9a990640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca9415f7d0 0x7fca9415fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9998e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fca9406ded0 0x7fca9406e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33406/0 (socket says 172.21.15.28:33406) 2024-04-25T19:11:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9998e640 1 -- 172.21.15.28:0/3296581370 learned_addr learned my addr 172.21.15.28:0/3296581370 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9a18f640 1 --2- 172.21.15.28:0/3296581370 >> v2:172.21.15.53:3300/0 conn(0x7fca940b5130 0x7fca94161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9a990640 1 -- 172.21.15.28:0/3296581370 >> v2:172.21.15.53:3300/0 conn(0x7fca940b5130 msgr2=0x7fca94161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9a990640 1 --2- 172.21.15.28:0/3296581370 >> v2:172.21.15.53:3300/0 conn(0x7fca940b5130 0x7fca94161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9a990640 1 -- 172.21.15.28:0/3296581370 >> v1:172.21.15.28:6789/0 conn(0x7fca9406ded0 legacy=0x7fca9406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.695+0000 7fca9a990640 1 -- 172.21.15.28:0/3296581370 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca94164160 con 0x7fca9415f7d0 2024-04-25T19:11:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.696+0000 7fca9a990640 1 --2- 172.21.15.28:0/3296581370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9415f7d0 0x7fca9415fbe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fca9406ec40 tx=0x7fca8402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b409da16957553de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.696+0000 7fca9918d640 1 -- 172.21.15.28:0/3296581370 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca84007740 con 0x7fca9415f7d0 2024-04-25T19:11:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.696+0000 7fca9918d640 1 -- 172.21.15.28:0/3296581370 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca840078e0 con 0x7fca9415f7d0 2024-04-25T19:11:39.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.696+0000 7fca9918d640 1 -- 172.21.15.28:0/3296581370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca84007cb0 con 0x7fca9415f7d0 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.697+0000 7fca9c3f3640 1 -- 172.21.15.28:0/3296581370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9415f7d0 msgr2=0x7fca9415fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.697+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/3296581370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9415f7d0 0x7fca9415fbe0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fca9406ec40 tx=0x7fca8402a680 comp rx=0 tx=0).stop 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.697+0000 7fca9c3f3640 1 -- 172.21.15.28:0/3296581370 shutdown_connections 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.697+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/3296581370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9415f7d0 0x7fca9415fbe0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.697+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/3296581370 >> v2:172.21.15.53:3300/0 conn(0x7fca940b5130 0x7fca94161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.697+0000 7fca9c3f3640 1 -- 172.21.15.28:0/3296581370 >> 172.21.15.28:0/3296581370 conn(0x7fca940b7ad0 msgr2=0x7fca940b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.697+0000 7fca9c3f3640 1 -- 172.21.15.28:0/3296581370 shutdown_connections 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.697+0000 7fca9c3f3640 1 -- 172.21.15.28:0/3296581370 wait complete. 2024-04-25T19:11:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.699+0000 7fca9c3f3640 1 Processor -- start 2024-04-25T19:11:39.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.699+0000 7fca9c3f3640 1 -- start start 2024-04-25T19:11:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9c3f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9406ded0 0x7fca940c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9c3f3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca940b5130 0x7fca9415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9c3f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca9415f7d0 0x7fca941f9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9c3f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca94160120 con 0x7fca9406ded0 2024-04-25T19:11:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9c3f3640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fca94164720 con 0x7fca940b5130 2024-04-25T19:11:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9c3f3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fca941602a0 con 0x7fca9415f7d0 2024-04-25T19:11:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a18f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9406ded0 0x7fca940c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a18f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9406ded0 0x7fca940c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36850/0 (socket says 172.21.15.28:36850) 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a18f640 1 -- 172.21.15.28:0/2429467343 learned_addr learned my addr 172.21.15.28:0/2429467343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9998e640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca940b5130 0x7fca9415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a990640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca9415f7d0 0x7fca941f9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a18f640 1 -- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca9415f7d0 msgr2=0x7fca941f9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a18f640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca9415f7d0 0x7fca941f9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a18f640 1 -- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca940b5130 msgr2=0x7fca9415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a18f640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca940b5130 0x7fca9415ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9a18f640 1 -- 172.21.15.28:0/2429467343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca941580d0 con 0x7fca9406ded0 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.700+0000 7fca9998e640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca940b5130 0x7fca9415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.701+0000 7fca9a18f640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9406ded0 0x7fca940c87b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00e870 tx=0x7fca8c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.701+0000 7fca92ffd640 1 -- 172.21.15.28:0/2429467343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c017070 con 0x7fca9406ded0 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.701+0000 7fca92ffd640 1 -- 172.21.15.28:0/2429467343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca8c00c8e0 con 0x7fca9406ded0 2024-04-25T19:11:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.701+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca94158e70 con 0x7fca9406ded0 2024-04-25T19:11:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.701+0000 7fca92ffd640 1 -- 172.21.15.28:0/2429467343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8c00ccb0 con 0x7fca9406ded0 2024-04-25T19:11:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.701+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca940b9dc0 con 0x7fca9406ded0 2024-04-25T19:11:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.702+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca94050740 con 0x7fca9406ded0 2024-04-25T19:11:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.703+0000 7fca92ffd640 1 -- 172.21.15.28:0/2429467343 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fca8c01a070 con 0x7fca9406ded0 2024-04-25T19:11:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.703+0000 7fca92ffd640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca800744b0 0x7fca800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.703+0000 7fca92ffd640 1 -- 172.21.15.28:0/2429467343 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fca8c095a70 con 0x7fca9406ded0 2024-04-25T19:11:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.703+0000 7fca9998e640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca800744b0 0x7fca800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.704+0000 7fca9998e640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca800744b0 0x7fca800769c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca940bbb80 tx=0x7fca88009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:39.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.707+0000 7fca92ffd640 1 -- 172.21.15.28:0/2429467343 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca8c060030 con 0x7fca9406ded0 2024-04-25T19:11:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.892+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca940d2fe0 con 0x7fca9406ded0 2024-04-25T19:11:39.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.894+0000 7fca92ffd640 1 -- 172.21.15.28:0/2429467343 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7fca8c05f930 con 0x7fca9406ded0 2024-04-25T19:11:39.896 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:39.896 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca800744b0 msgr2=0x7fca800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca800744b0 0x7fca800769c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fca940bbb80 tx=0x7fca88009040 comp rx=0 tx=0).stop 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9406ded0 msgr2=0x7fca940c87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9406ded0 0x7fca940c87b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fca8c00e870 tx=0x7fca8c00ed30 comp rx=0 tx=0).stop 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 shutdown_connections 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca800744b0 0x7fca800769c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca9415f7d0 0x7fca941f9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca940b5130 0x7fca9415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 --2- 172.21.15.28:0/2429467343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9406ded0 0x7fca940c87b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 >> 172.21.15.28:0/2429467343 conn(0x7fca940b7ad0 msgr2=0x7fca94161610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 shutdown_connections 2024-04-25T19:11:39.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:39.896+0000 7fca9c3f3640 1 -- 172.21.15.28:0/2429467343 wait complete. 2024-04-25T19:11:39.909 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:11:39.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:11:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.022+0000 7fb214a30640 1 Processor -- start 2024-04-25T19:11:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.022+0000 7fb214a30640 1 -- start start 2024-04-25T19:11:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.023+0000 7fb214a30640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb210160120 0x7fb2100b7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.023+0000 7fb214a30640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb2100b7e80 0x7fb210160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.023+0000 7fb214a30640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb210160530 con 0x7fb210160cc0 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.023+0000 7fb214a30640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb210164580 con 0x7fb210160120 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.023+0000 7fb214a30640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fb2100d5460 con 0x7fb2100b7e80 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.024+0000 7fb20e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb210160120 0x7fb2100b7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.024+0000 7fb20e59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb210160120 0x7fb2100b7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36864/0 (socket says 172.21.15.28:36864) 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.024+0000 7fb20e59c640 1 -- 172.21.15.28:0/44547341 learned_addr learned my addr 172.21.15.28:0/44547341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.024+0000 7fb20dd9b640 1 --2- 172.21.15.28:0/44547341 >> v2:172.21.15.53:3300/0 conn(0x7fb2100b7e80 0x7fb210160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.024+0000 7fb20e59c640 1 -- 172.21.15.28:0/44547341 >> v2:172.21.15.53:3300/0 conn(0x7fb2100b7e80 msgr2=0x7fb210160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.024+0000 7fb20e59c640 1 --2- 172.21.15.28:0/44547341 >> v2:172.21.15.53:3300/0 conn(0x7fb2100b7e80 0x7fb210160780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.024+0000 7fb20e59c640 1 -- 172.21.15.28:0/44547341 >> v1:172.21.15.53:6789/0 conn(0x7fb210160cc0 legacy=0x7fb2100d0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.024+0000 7fb20e59c640 1 -- 172.21.15.28:0/44547341 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2100b8290 con 0x7fb210160120 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.025+0000 7fb20e59c640 1 --2- 172.21.15.28:0/44547341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb210160120 0x7fb2100b7940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb21008efd0 tx=0x7fb20802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=380fedfbdb637ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.025+0000 7fb20d59a640 1 -- 172.21.15.28:0/44547341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208037070 con 0x7fb210160120 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.025+0000 7fb20d59a640 1 -- 172.21.15.28:0/44547341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb208032440 con 0x7fb210160120 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.025+0000 7fb20d59a640 1 -- 172.21.15.28:0/44547341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208032810 con 0x7fb210160120 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.025+0000 7fb214a30640 1 -- 172.21.15.28:0/44547341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb210160120 msgr2=0x7fb2100b7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.026+0000 7fb214a30640 1 --2- 172.21.15.28:0/44547341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb210160120 0x7fb2100b7940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb21008efd0 tx=0x7fb20802a270 comp rx=0 tx=0).stop 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.026+0000 7fb214a30640 1 -- 172.21.15.28:0/44547341 shutdown_connections 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.026+0000 7fb214a30640 1 --2- 172.21.15.28:0/44547341 >> v2:172.21.15.53:3300/0 conn(0x7fb2100b7e80 0x7fb210160780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.026+0000 7fb214a30640 1 --2- 172.21.15.28:0/44547341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb210160120 0x7fb2100b7940 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.026+0000 7fb214a30640 1 -- 172.21.15.28:0/44547341 >> 172.21.15.28:0/44547341 conn(0x7fb2100d2980 msgr2=0x7fb2100d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.026+0000 7fb214a30640 1 -- 172.21.15.28:0/44547341 shutdown_connections 2024-04-25T19:11:40.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.026+0000 7fb214a30640 1 -- 172.21.15.28:0/44547341 wait complete. 2024-04-25T19:11:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.028+0000 7fb214a30640 1 Processor -- start 2024-04-25T19:11:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.028+0000 7fb214a30640 1 -- start start 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb214a30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2100b7e80 0x7fb2101ebf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb214a30640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb210160120 0x7fb2101f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb214a30640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb210160cc0 0x7fb210158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb214a30640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb210163f30 con 0x7fb2100b7e80 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb214a30640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb210163c50 con 0x7fb210160cc0 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb214a30640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb210164210 con 0x7fb210160120 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2100b7e80 0x7fb2101ebf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2100b7e80 0x7fb2101ebf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36880/0 (socket says 172.21.15.28:36880) 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 -- 172.21.15.28:0/844355772 learned_addr learned my addr 172.21.15.28:0/844355772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 -- 172.21.15.28:0/844355772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb210160120 msgr2=0x7fb2101f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb210160120 0x7fb2101f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 -- 172.21.15.28:0/844355772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb210160cc0 msgr2=0x7fb210158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20ed9d640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb210160cc0 0x7fb210158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb210160cc0 0x7fb210158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 -- 172.21.15.28:0/844355772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2101f75b0 con 0x7fb2100b7e80 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.029+0000 7fb20e59c640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2100b7e80 0x7fb2101ebf00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb20802a2b0 tx=0x7fb2080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.030+0000 7fb1feffd640 1 -- 172.21.15.28:0/844355772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208041020 con 0x7fb2100b7e80 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.030+0000 7fb1feffd640 1 -- 172.21.15.28:0/844355772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2080306c0 con 0x7fb2100b7e80 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.030+0000 7fb1feffd640 1 -- 172.21.15.28:0/844355772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208037040 con 0x7fb2100b7e80 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.030+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2101eae60 con 0x7fb2100b7e80 2024-04-25T19:11:40.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.030+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2101eb420 con 0x7fb2100b7e80 2024-04-25T19:11:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.031+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb210050740 con 0x7fb2100b7e80 2024-04-25T19:11:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.031+0000 7fb1feffd640 1 -- 172.21.15.28:0/844355772 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb208044020 con 0x7fb2100b7e80 2024-04-25T19:11:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.032+0000 7fb1feffd640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1f00744b0 0x7fb1f00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.032+0000 7fb20dd9b640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1f00744b0 0x7fb1f00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.032+0000 7fb1feffd640 1 -- 172.21.15.28:0/844355772 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb2080b5e00 con 0x7fb2100b7e80 2024-04-25T19:11:40.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.032+0000 7fb20dd9b640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1f00744b0 0x7fb1f00769c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb204006fd0 tx=0x7fb204008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.037+0000 7fb1feffd640 1 -- 172.21.15.28:0/844355772 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb20804b030 con 0x7fb2100b7e80 2024-04-25T19:11:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.220+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb21015f510 con 0x7fb2100b7e80 2024-04-25T19:11:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.220+0000 7fb1feffd640 1 -- 172.21.15.28:0/844355772 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fb2101f75b0 con 0x7fb2100b7e80 2024-04-25T19:11:40.222 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:11:40.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1f00744b0 msgr2=0x7fb1f00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1f00744b0 0x7fb1f00769c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb204006fd0 tx=0x7fb204008040 comp rx=0 tx=0).stop 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2100b7e80 msgr2=0x7fb2101ebf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2100b7e80 0x7fb2101ebf00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb20802a2b0 tx=0x7fb2080046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 shutdown_connections 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1f00744b0 0x7fb1f00769c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb210160cc0 0x7fb210158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb210160120 0x7fb2101f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 --2- 172.21.15.28:0/844355772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2100b7e80 0x7fb2101ebf00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 >> 172.21.15.28:0/844355772 conn(0x7fb2100d2980 msgr2=0x7fb2100d6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 shutdown_connections 2024-04-25T19:11:40.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.222+0000 7fb214a30640 1 -- 172.21.15.28:0/844355772 wait complete. 2024-04-25T19:11:40.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:11:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.348+0000 7f7dd1396640 1 Processor -- start 2024-04-25T19:11:40.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.348+0000 7f7dd1396640 1 -- start start 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.349+0000 7f7dd1396640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7dcc0d0ee0 0x7f7dcc0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.349+0000 7f7dd1396640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 0x7f7dcc160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.349+0000 7f7dd1396640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dcc06f5b0 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.349+0000 7f7dd1396640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f7dcc0af290 con 0x7f7dcc0d0ee0 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.349+0000 7f7dd1396640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7dcc0d14a0 con 0x7f7dcc0b5020 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dc3fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7dcc0d0ee0 0x7f7dcc0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dc3fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7dcc0d0ee0 0x7f7dcc0affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36886/0 (socket says 172.21.15.28:36886) 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dc3fff640 1 -- 172.21.15.28:0/724179524 learned_addr learned my addr 172.21.15.28:0/724179524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dcb7fe640 1 --2- 172.21.15.28:0/724179524 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 0x7f7dcc160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dc3fff640 1 -- 172.21.15.28:0/724179524 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 msgr2=0x7f7dcc160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dc3fff640 1 --2- 172.21.15.28:0/724179524 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 0x7f7dcc160990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dca7fc640 1 -- 172.21.15.28:0/724179524 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3024223409 0 0) 0x7f7dcc06f5b0 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dc3fff640 1 -- 172.21.15.28:0/724179524 >> v1:172.21.15.53:6789/0 conn(0x7f7dcc15aaa0 legacy=0x7f7dcc0b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.350+0000 7f7dc3fff640 1 -- 172.21.15.28:0/724179524 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dcc0af4e0 con 0x7f7dcc0d0ee0 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.351+0000 7f7dc3fff640 1 --2- 172.21.15.28:0/724179524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc0d0ee0 0x7f7dcc0affc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7db400bb10 tx=0x7f7db402c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455adf640c960ca5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.351+0000 7f7dca7fc640 1 -- 172.21.15.28:0/724179524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4036020 con 0x7f7dcc0d0ee0 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.351+0000 7f7dca7fc640 1 -- 172.21.15.28:0/724179524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7db4032820 con 0x7f7dcc0d0ee0 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.351+0000 7f7dca7fc640 1 -- 172.21.15.28:0/724179524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db4032bf0 con 0x7f7dcc0d0ee0 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.351+0000 7f7dd1396640 1 -- 172.21.15.28:0/724179524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc0d0ee0 msgr2=0x7f7dcc0affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.351+0000 7f7dd1396640 1 --2- 172.21.15.28:0/724179524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc0d0ee0 0x7f7dcc0affc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7db400bb10 tx=0x7f7db402c920 comp rx=0 tx=0).stop 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.352+0000 7f7dd1396640 1 -- 172.21.15.28:0/724179524 shutdown_connections 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.352+0000 7f7dd1396640 1 --2- 172.21.15.28:0/724179524 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 0x7f7dcc160990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.352+0000 7f7dd1396640 1 --2- 172.21.15.28:0/724179524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc0d0ee0 0x7f7dcc0affc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.352+0000 7f7dd1396640 1 -- 172.21.15.28:0/724179524 >> 172.21.15.28:0/724179524 conn(0x7f7dcc0742c0 msgr2=0x7f7dcc06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.352+0000 7f7dd1396640 1 -- 172.21.15.28:0/724179524 shutdown_connections 2024-04-25T19:11:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.352+0000 7f7dd1396640 1 -- 172.21.15.28:0/724179524 wait complete. 2024-04-25T19:11:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.353+0000 7f7dd1396640 1 Processor -- start 2024-04-25T19:11:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.353+0000 7f7dd1396640 1 -- start start 2024-04-25T19:11:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dd1396640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 0x7f7dcc0b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dd1396640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0d0ee0 0x7f7dcc0b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dd1396640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc15aaa0 0x7f7dcc154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dd1396640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc0b5430 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dd1396640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc0afaa0 con 0x7f7dcc0d0ee0 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dd1396640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7dcc0b55b0 con 0x7f7dcc0b5020 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc15aaa0 0x7f7dcc154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc15aaa0 0x7f7dcc154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36898/0 (socket says 172.21.15.28:36898) 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1725418245 learned_addr learned my addr 172.21.15.28:0/1725418245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dc3fff640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0d0ee0 0x7f7dcc0b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcaffd640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 0x7f7dcc0b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1725418245 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 msgr2=0x7f7dcc0b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 0x7f7dcc0b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1725418245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0d0ee0 msgr2=0x7f7dcc0b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0d0ee0 0x7f7dcc0b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 -- 172.21.15.28:0/1725418245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dcc1f3950 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dcb7fe640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc15aaa0 0x7f7dcc154ba0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400efc0 tx=0x7f7dc400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dc37fe640 1 -- 172.21.15.28:0/1725418245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4018020 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dc37fe640 1 -- 172.21.15.28:0/1725418245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc4014480 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dcc1597d0 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dc37fe640 1 -- 172.21.15.28:0/1725418245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4009e60 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.355+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dcc0b1ae0 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.357+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dcc050740 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.357+0000 7f7dc37fe640 1 -- 172.21.15.28:0/1725418245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7dc4005d10 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.358+0000 7f7dc37fe640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dac0744b0 0x7f7dac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.358+0000 7f7dc37fe640 1 -- 172.21.15.28:0/1725418245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7dc4096300 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.358+0000 7f7dcaffd640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dac0744b0 0x7f7dac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.358+0000 7f7dcaffd640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dac0744b0 0x7f7dac0769c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc007fd0 tx=0x7f7dbc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.362+0000 7f7dc37fe640 1 -- 172.21.15.28:0/1725418245 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7dc4022030 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.546+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7dcc154960 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.546+0000 7f7dc37fe640 1 -- 172.21.15.28:0/1725418245 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f7dc4010040 con 0x7f7dcc15aaa0 2024-04-25T19:11:40.548 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:11:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dac0744b0 msgr2=0x7f7dac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dac0744b0 0x7f7dac0769c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc007fd0 tx=0x7f7dbc009040 comp rx=0 tx=0).stop 2024-04-25T19:11:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc15aaa0 msgr2=0x7f7dcc154ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc15aaa0 0x7f7dcc154ba0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400efc0 tx=0x7f7dc400c490 comp rx=0 tx=0).stop 2024-04-25T19:11:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 shutdown_connections 2024-04-25T19:11:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dac0744b0 0x7f7dac0769c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dcc15aaa0 0x7f7dcc154ba0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dcc0d0ee0 0x7f7dcc0b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 --2- 172.21.15.28:0/1725418245 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dcc0b5020 0x7f7dcc0b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 >> 172.21.15.28:0/1725418245 conn(0x7f7dcc0742c0 msgr2=0x7f7dcc0cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 shutdown_connections 2024-04-25T19:11:40.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.548+0000 7f7dd1396640 1 -- 172.21.15.28:0/1725418245 wait complete. 2024-04-25T19:11:40.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:11:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.673+0000 7f62fb642640 1 Processor -- start 2024-04-25T19:11:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.674+0000 7f62fb642640 1 -- start start 2024-04-25T19:11:40.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62fb642640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f62f40b4c80 0x7f62f4160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62fb642640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62f40b7fd0 0x7f62f40b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62fb642640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62f40d15b0 con 0x7f62f4164800 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62fb642640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f62f4164c10 con 0x7f62f40b7fd0 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62fb642640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f62f40d12d0 con 0x7f62f40b4c80 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62f9bdf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62f40b7fd0 0x7f62f40b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62f9bdf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62f40b7fd0 0x7f62f40b46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46878/0 (socket says 172.21.15.28:46878) 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62f9bdf640 1 -- 172.21.15.28:0/2912267938 learned_addr learned my addr 172.21.15.28:0/2912267938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62f93de640 1 --2- 172.21.15.28:0/2912267938 >> v2:172.21.15.53:3300/0 conn(0x7f62f40b4c80 0x7f62f4160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62f9bdf640 1 -- 172.21.15.28:0/2912267938 >> v2:172.21.15.53:3300/0 conn(0x7f62f40b4c80 msgr2=0x7f62f4160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62f9bdf640 1 --2- 172.21.15.28:0/2912267938 >> v2:172.21.15.53:3300/0 conn(0x7f62f40b4c80 0x7f62f4160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62f9bdf640 1 -- 172.21.15.28:0/2912267938 >> v1:172.21.15.53:6789/0 conn(0x7f62f4164800 legacy=0x7f62f40b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:40.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.675+0000 7f62f9bdf640 1 -- 172.21.15.28:0/2912267938 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f4163d20 con 0x7f62f40b7fd0 2024-04-25T19:11:40.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.676+0000 7f62f9bdf640 1 --2- 172.21.15.28:0/2912267938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f40b7fd0 0x7f62f40b46c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f62f40d3840 tx=0x7f62f002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55a1f952a368a7c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.676+0000 7f62ebfff640 1 -- 172.21.15.28:0/2912267938 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0036070 con 0x7f62f40b7fd0 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.676+0000 7f62ebfff640 1 -- 172.21.15.28:0/2912267938 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f0031600 con 0x7f62f40b7fd0 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.676+0000 7f62ebfff640 1 -- 172.21.15.28:0/2912267938 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f00319d0 con 0x7f62f40b7fd0 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.677+0000 7f62fb642640 1 -- 172.21.15.28:0/2912267938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f40b7fd0 msgr2=0x7f62f40b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.677+0000 7f62fb642640 1 --2- 172.21.15.28:0/2912267938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f40b7fd0 0x7f62f40b46c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f62f40d3840 tx=0x7f62f002a6f0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.677+0000 7f62fb642640 1 -- 172.21.15.28:0/2912267938 shutdown_connections 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.677+0000 7f62fb642640 1 --2- 172.21.15.28:0/2912267938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f40b7fd0 0x7f62f40b46c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.677+0000 7f62fb642640 1 --2- 172.21.15.28:0/2912267938 >> v2:172.21.15.53:3300/0 conn(0x7f62f40b4c80 0x7f62f4160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.677+0000 7f62fb642640 1 -- 172.21.15.28:0/2912267938 >> 172.21.15.28:0/2912267938 conn(0x7f62f40d2980 msgr2=0x7f62f40d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.677+0000 7f62fb642640 1 -- 172.21.15.28:0/2912267938 shutdown_connections 2024-04-25T19:11:40.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.677+0000 7f62fb642640 1 -- 172.21.15.28:0/2912267938 wait complete. 2024-04-25T19:11:40.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.679+0000 7f62fb642640 1 Processor -- start 2024-04-25T19:11:40.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.679+0000 7f62fb642640 1 -- start start 2024-04-25T19:11:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.680+0000 7f62fb642640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62f40b4c80 0x7f62f4154420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.680+0000 7f62fb642640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62f4164800 0x7f62f415e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.680+0000 7f62fb642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f4154a60 0x7f62f415ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.680+0000 7f62fb642640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62f40b5240 con 0x7f62f4154a60 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.680+0000 7f62fb642640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f62f41642e0 con 0x7f62f40b4c80 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.680+0000 7f62fb642640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f62f40b53c0 con 0x7f62f4164800 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f4154a60 0x7f62f415ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f4154a60 0x7f62f415ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46886/0 (socket says 172.21.15.28:46886) 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 -- 172.21.15.28:0/2020322452 learned_addr learned my addr 172.21.15.28:0/2020322452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f93de640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62f40b4c80 0x7f62f4154420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f8bdd640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62f4164800 0x7f62f415e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 -- 172.21.15.28:0/2020322452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62f4164800 msgr2=0x7f62f415e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62f4164800 0x7f62f415e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 -- 172.21.15.28:0/2020322452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62f40b4c80 msgr2=0x7f62f4154420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62f40b4c80 0x7f62f4154420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 -- 172.21.15.28:0/2020322452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62f415ad60 con 0x7f62f4154a60 2024-04-25T19:11:40.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62f9bdf640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f4154a60 0x7f62f415ee30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f62f002ace0 tx=0x7f62f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62e9ffb640 1 -- 172.21.15.28:0/2020322452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0036050 con 0x7f62f4154a60 2024-04-25T19:11:40.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62e9ffb640 1 -- 172.21.15.28:0/2020322452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f0042070 con 0x7f62f4154a60 2024-04-25T19:11:40.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62f415b040 con 0x7f62f4154a60 2024-04-25T19:11:40.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.681+0000 7f62e9ffb640 1 -- 172.21.15.28:0/2020322452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f0009510 con 0x7f62f4154a60 2024-04-25T19:11:40.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.682+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62f40cb260 con 0x7f62f4154a60 2024-04-25T19:11:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.682+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62f4050740 con 0x7f62f4154a60 2024-04-25T19:11:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.683+0000 7f62e9ffb640 1 -- 172.21.15.28:0/2020322452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f62f0007c30 con 0x7f62f4154a60 2024-04-25T19:11:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.684+0000 7f62e9ffb640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62d80744b0 0x7f62d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.684+0000 7f62f93de640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62d80744b0 0x7f62d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.684+0000 7f62e9ffb640 1 -- 172.21.15.28:0/2020322452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f62f00b6310 con 0x7f62f4154a60 2024-04-25T19:11:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.684+0000 7f62f93de640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62d80744b0 0x7f62d80769c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0079a0 tx=0x7f62ec008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:40.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.688+0000 7f62e9ffb640 1 -- 172.21.15.28:0/2020322452 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62f00801a0 con 0x7f62f4154a60 2024-04-25T19:11:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.874+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f62f40d3f70 con 0x7f62f4154a60 2024-04-25T19:11:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.874+0000 7f62e9ffb640 1 -- 172.21.15.28:0/2020322452 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f62f00844a0 con 0x7f62f4154a60 2024-04-25T19:11:40.876 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:11:40.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62d80744b0 msgr2=0x7f62d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62d80744b0 0x7f62d80769c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f62ec0079a0 tx=0x7f62ec008040 comp rx=0 tx=0).stop 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f4154a60 msgr2=0x7f62f415ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f4154a60 0x7f62f415ee30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f62f002ace0 tx=0x7f62f0004a40 comp rx=0 tx=0).stop 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 shutdown_connections 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62d80744b0 0x7f62d80769c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f4154a60 0x7f62f415ee30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62f4164800 0x7f62f415e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 --2- 172.21.15.28:0/2020322452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62f40b4c80 0x7f62f4154420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 >> 172.21.15.28:0/2020322452 conn(0x7f62f40d2980 msgr2=0x7f62f40d52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.876+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 shutdown_connections 2024-04-25T19:11:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:40.877+0000 7f62fb642640 1 -- 172.21.15.28:0/2020322452 wait complete. 2024-04-25T19:11:40.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.003+0000 7fd0b2a51640 1 Processor -- start 2024-04-25T19:11:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.003+0000 7fd0b2a51640 1 -- start start 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.004+0000 7fd0b2a51640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0ac0b30e0 0x7fd0ac0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.004+0000 7fd0b2a51640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac0b3650 con 0x7fd0ac09b540 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.004+0000 7fd0b2a51640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac06c710 con 0x7fd0ac06d1f0 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.004+0000 7fd0b2a51640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fd0ac06c9f0 con 0x7fd0ac0b30e0 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.005+0000 7fd0abfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0ac0b30e0 0x7fd0ac0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.005+0000 7fd0ab7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd0ac09b540 0x7fd0ac09b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52332/0 (socket says 172.21.15.28:52332) 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.005+0000 7fd0ab7fe640 1 -- 172.21.15.28:0/1376558755 learned_addr learned my addr 172.21.15.28:0/1376558755 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.005+0000 7fd0abfff640 1 -- 172.21.15.28:0/1376558755 >> v1:172.21.15.53:6789/0 conn(0x7fd0ac06d1f0 legacy=0x7fd0ac06d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.005+0000 7fd0abfff640 1 -- 172.21.15.28:0/1376558755 >> v1:172.21.15.28:6789/0 conn(0x7fd0ac09b540 legacy=0x7fd0ac09b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.005+0000 7fd0aaffd640 1 -- 172.21.15.28:0/1376558755 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2469860634 0 0) 0x7fd0ac0b3650 con 0x7fd0ac09b540 2024-04-25T19:11:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.005+0000 7fd0abfff640 1 -- 172.21.15.28:0/1376558755 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac06ccd0 con 0x7fd0ac0b30e0 2024-04-25T19:11:41.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0abfff640 1 --2- 172.21.15.28:0/1376558755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac0b30e0 0x7fd0ac0cf7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0a00097a0 tx=0x7fd0a002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2e256d028066b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0aaffd640 1 -- 172.21.15.28:0/1376558755 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0037070 con 0x7fd0ac0b30e0 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0aaffd640 1 -- 172.21.15.28:0/1376558755 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a0032440 con 0x7fd0ac0b30e0 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0aaffd640 1 -- 172.21.15.28:0/1376558755 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0032810 con 0x7fd0ac0b30e0 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0b2a51640 1 -- 172.21.15.28:0/1376558755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac0b30e0 msgr2=0x7fd0ac0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0b2a51640 1 --2- 172.21.15.28:0/1376558755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac0b30e0 0x7fd0ac0cf7d0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0a00097a0 tx=0x7fd0a002a1a0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0b2a51640 1 -- 172.21.15.28:0/1376558755 shutdown_connections 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0b2a51640 1 --2- 172.21.15.28:0/1376558755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac0b30e0 0x7fd0ac0cf7d0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.006+0000 7fd0b2a51640 1 -- 172.21.15.28:0/1376558755 >> 172.21.15.28:0/1376558755 conn(0x7fd0ac0b7620 msgr2=0x7fd0ac0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.007+0000 7fd0b2a51640 1 -- 172.21.15.28:0/1376558755 shutdown_connections 2024-04-25T19:11:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.007+0000 7fd0b2a51640 1 -- 172.21.15.28:0/1376558755 wait complete. 2024-04-25T19:11:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.008+0000 7fd0b2a51640 1 Processor -- start 2024-04-25T19:11:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.008+0000 7fd0b2a51640 1 -- start start 2024-04-25T19:11:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0b2a51640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0ac06d1f0 0x7fd0ac1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0b2a51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac09b540 0x7fd0ac152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0b2a51640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0ac0b30e0 0x7fd0ac158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0b2a51640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac161ac0 con 0x7fd0ac09b540 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0b2a51640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac161940 con 0x7fd0ac06d1f0 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0b2a51640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd0ac06dd10 con 0x7fd0ac0b30e0 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac09b540 0x7fd0ac152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac09b540 0x7fd0ac152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46918/0 (socket says 172.21.15.28:46918) 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 -- 172.21.15.28:0/77557637 learned_addr learned my addr 172.21.15.28:0/77557637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0abfff640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0ac06d1f0 0x7fd0ac1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0b0fee640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0ac0b30e0 0x7fd0ac158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 -- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0ac0b30e0 msgr2=0x7fd0ac158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0ac0b30e0 0x7fd0ac158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 -- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0ac06d1f0 msgr2=0x7fd0ac1560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0ac06d1f0 0x7fd0ac1560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 -- 172.21.15.28:0/77557637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac156a30 con 0x7fd0ac09b540 2024-04-25T19:11:41.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0b0fee640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0ac0b30e0 0x7fd0ac158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.010+0000 7fd0ab7fe640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac09b540 0x7fd0ac152290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00c3f0 tx=0x7fd09c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.011+0000 7fd0a8ff9640 1 -- 172.21.15.28:0/77557637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0074e0 con 0x7fd0ac09b540 2024-04-25T19:11:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.011+0000 7fd0a8ff9640 1 -- 172.21.15.28:0/77557637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd09c0047e0 con 0x7fd0ac09b540 2024-04-25T19:11:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.011+0000 7fd0a8ff9640 1 -- 172.21.15.28:0/77557637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09c004bd0 con 0x7fd0ac09b540 2024-04-25T19:11:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.011+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ac156d10 con 0x7fd0ac09b540 2024-04-25T19:11:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.011+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0ac1572d0 con 0x7fd0ac09b540 2024-04-25T19:11:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.012+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0ac1567a0 con 0x7fd0ac09b540 2024-04-25T19:11:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.012+0000 7fd0a8ff9640 1 -- 172.21.15.28:0/77557637 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd09c011070 con 0x7fd0ac09b540 2024-04-25T19:11:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.013+0000 7fd0a8ff9640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0980744b0 0x7fd0980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.013+0000 7fd0a8ff9640 1 -- 172.21.15.28:0/77557637 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd09c095240 con 0x7fd0ac09b540 2024-04-25T19:11:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.013+0000 7fd0abfff640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0980744b0 0x7fd0980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.013+0000 7fd0abfff640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0980744b0 0x7fd0980769c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0034000 tx=0x7fd0a002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.018+0000 7fd0a8ff9640 1 -- 172.21.15.28:0/77557637 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd09c05f5f0 con 0x7fd0ac09b540 2024-04-25T19:11:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.206+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd0ac050740 con 0x7fd0ac09b540 2024-04-25T19:11:41.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.207+0000 7fd0a8ff9640 1 -- 172.21.15.28:0/77557637 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fd09c05eef0 con 0x7fd0ac09b540 2024-04-25T19:11:41.209 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0980744b0 msgr2=0x7fd0980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0980744b0 0x7fd0980769c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd0a0034000 tx=0x7fd0a002e000 comp rx=0 tx=0).stop 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac09b540 msgr2=0x7fd0ac152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac09b540 0x7fd0ac152290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00c3f0 tx=0x7fd09c00c8b0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 shutdown_connections 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0980744b0 0x7fd0980769c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd0ac0b30e0 0x7fd0ac158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0ac09b540 0x7fd0ac152290 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 --2- 172.21.15.28:0/77557637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0ac06d1f0 0x7fd0ac1560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 >> 172.21.15.28:0/77557637 conn(0x7fd0ac0b7620 msgr2=0x7fd0ac06d600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 shutdown_connections 2024-04-25T19:11:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.209+0000 7fd0b2a51640 1 -- 172.21.15.28:0/77557637 wait complete. 2024-04-25T19:11:41.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.333+0000 7fe3a9b7d640 1 Processor -- start 2024-04-25T19:11:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.333+0000 7fe3a9b7d640 1 -- start start 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.334+0000 7fe3a9b7d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3a40b81a0 0x7fe3a40d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.334+0000 7fe3a9b7d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe3a40d2ad0 0x7fe3a40d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.334+0000 7fe3a9b7d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3a40d2ee0 con 0x7fe3a4160250 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.334+0000 7fe3a9b7d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fe3a40b4f50 con 0x7fe3a40b81a0 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.334+0000 7fe3a9b7d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fe3a4164b20 con 0x7fe3a40d2ad0 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a2ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3a40b81a0 0x7fe3a40d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a37fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe3a4160250 0x7fe3a40b4690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:42480/0 (socket says 172.21.15.28:42480) 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a37fe640 1 -- 172.21.15.28:0/2045587076 learned_addr learned my addr 172.21.15.28:0/2045587076 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a2ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3a40b81a0 0x7fe3a40d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46928/0 (socket says 172.21.15.28:46928) 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a3fff640 1 --2- 172.21.15.28:0/2045587076 >> v2:172.21.15.53:3300/0 conn(0x7fe3a40d2ad0 0x7fe3a40d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a2ffd640 1 -- 172.21.15.28:0/2045587076 >> v2:172.21.15.53:3300/0 conn(0x7fe3a40d2ad0 msgr2=0x7fe3a40d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a2ffd640 1 --2- 172.21.15.28:0/2045587076 >> v2:172.21.15.53:3300/0 conn(0x7fe3a40d2ad0 0x7fe3a40d44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a2ffd640 1 -- 172.21.15.28:0/2045587076 >> v1:172.21.15.53:6789/0 conn(0x7fe3a4160250 legacy=0x7fe3a40b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a27fc640 1 -- 172.21.15.28:0/2045587076 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3622126223 0 0) 0x7fe394001110 con 0x7fe3a4160250 2024-04-25T19:11:41.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.335+0000 7fe3a2ffd640 1 -- 172.21.15.28:0/2045587076 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a4163d60 con 0x7fe3a40b81a0 2024-04-25T19:11:41.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.336+0000 7fe3a2ffd640 1 --2- 172.21.15.28:0/2045587076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a40d2590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40d3890 tx=0x7fe39802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35559978b84801fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.336+0000 7fe3a27fc640 1 -- 172.21.15.28:0/2045587076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398037070 con 0x7fe3a40b81a0 2024-04-25T19:11:41.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.336+0000 7fe3a27fc640 1 -- 172.21.15.28:0/2045587076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe398032440 con 0x7fe3a40b81a0 2024-04-25T19:11:41.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.336+0000 7fe3a27fc640 1 -- 172.21.15.28:0/2045587076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe39802ae60 con 0x7fe3a40b81a0 2024-04-25T19:11:41.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.337+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/2045587076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 msgr2=0x7fe3a40d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.337+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/2045587076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a40d2590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe3a40d3890 tx=0x7fe39802a110 comp rx=0 tx=0).stop 2024-04-25T19:11:41.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.337+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/2045587076 shutdown_connections 2024-04-25T19:11:41.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.337+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/2045587076 >> v2:172.21.15.53:3300/0 conn(0x7fe3a40d2ad0 0x7fe3a40d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.337+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/2045587076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a40d2590 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.337+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/2045587076 >> 172.21.15.28:0/2045587076 conn(0x7fe3a406ee30 msgr2=0x7fe3a406c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.337+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/2045587076 shutdown_connections 2024-04-25T19:11:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.337+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/2045587076 wait complete. 2024-04-25T19:11:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.338+0000 7fe3a9b7d640 1 Processor -- start 2024-04-25T19:11:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.339+0000 7fe3a9b7d640 1 -- start start 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a9b7d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a415a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a9b7d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3a40d2ad0 0x7fe3a4156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a9b7d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3a4160250 0x7fe3a415ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a9b7d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a4164600 con 0x7fe3a40b81a0 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a9b7d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a4164320 con 0x7fe3a4160250 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a9b7d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a40d49f0 con 0x7fe3a40d2ad0 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a415a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a415a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46936/0 (socket says 172.21.15.28:46936) 2024-04-25T19:11:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a37fe640 1 -- 172.21.15.28:0/654759365 learned_addr learned my addr 172.21.15.28:0/654759365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a3fff640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3a4160250 0x7fe3a415ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a2ffd640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3a40d2ad0 0x7fe3a4156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a37fe640 1 -- 172.21.15.28:0/654759365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3a40d2ad0 msgr2=0x7fe3a4156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a37fe640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3a40d2ad0 0x7fe3a4156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a37fe640 1 -- 172.21.15.28:0/654759365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3a4160250 msgr2=0x7fe3a415ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a37fe640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3a4160250 0x7fe3a415ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.340+0000 7fe3a37fe640 1 -- 172.21.15.28:0/654759365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a41e9a50 con 0x7fe3a40b81a0 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.341+0000 7fe3a37fe640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a415a700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c440 tx=0x7fe39400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.341+0000 7fe38bfff640 1 -- 172.21.15.28:0/654759365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394018070 con 0x7fe3a40b81a0 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.341+0000 7fe38bfff640 1 -- 172.21.15.28:0/654759365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3940074e0 con 0x7fe3a40b81a0 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.341+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a41e9cc0 con 0x7fe3a40b81a0 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.341+0000 7fe38bfff640 1 -- 172.21.15.28:0/654759365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394004a20 con 0x7fe3a40b81a0 2024-04-25T19:11:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.341+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a41ea280 con 0x7fe3a40b81a0 2024-04-25T19:11:41.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.342+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a4050740 con 0x7fe3a40b81a0 2024-04-25T19:11:41.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.343+0000 7fe38bfff640 1 -- 172.21.15.28:0/654759365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe39401b070 con 0x7fe3a40b81a0 2024-04-25T19:11:41.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.343+0000 7fe38bfff640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe3900744e0 0x7fe3900769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.343+0000 7fe3a2ffd640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe3900744e0 0x7fe3900769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.343+0000 7fe38bfff640 1 -- 172.21.15.28:0/654759365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe3940946e0 con 0x7fe3a40b81a0 2024-04-25T19:11:41.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.344+0000 7fe3a2ffd640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe3900744e0 0x7fe3900769f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe398034000 tx=0x7fe39802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.348+0000 7fe38bfff640 1 -- 172.21.15.28:0/654759365 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3a40d2ee0 con 0x7fe3a40b81a0 2024-04-25T19:11:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.543+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe3a40d3fd0 con 0x7fe3a40b81a0 2024-04-25T19:11:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.543+0000 7fe38bfff640 1 -- 172.21.15.28:0/654759365 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7fe39405e4e0 con 0x7fe3a40b81a0 2024-04-25T19:11:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-04-25T19:11:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe3900744e0 msgr2=0x7fe3900769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe3900744e0 0x7fe3900769f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe398034000 tx=0x7fe39802e000 comp rx=0 tx=0).stop 2024-04-25T19:11:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 msgr2=0x7fe3a415a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a415a700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c440 tx=0x7fe39400c900 comp rx=0 tx=0).stop 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 shutdown_connections 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe3900744e0 0x7fe3900769f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3a4160250 0x7fe3a415ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3a40d2ad0 0x7fe3a4156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 --2- 172.21.15.28:0/654759365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3a40b81a0 0x7fe3a415a700 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 >> 172.21.15.28:0/654759365 conn(0x7fe3a406ee30 msgr2=0x7fe3a415e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 shutdown_connections 2024-04-25T19:11:41.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.545+0000 7fe3a9b7d640 1 -- 172.21.15.28:0/654759365 wait complete. 2024-04-25T19:11:41.558 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:11:41.558 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:11:41.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.723+0000 7f465fa6e640 1 Processor -- start 2024-04-25T19:11:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.724+0000 7f465fa6e640 1 -- start start 2024-04-25T19:11:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465fa6e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46580b91a0 0x7f4658163850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465fa6e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46580d1d30 con 0x7f46580d2ab0 2024-04-25T19:11:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465fa6e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465806ebb0 con 0x7f465809cb20 2024-04-25T19:11:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465fa6e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f46580d2620 con 0x7f46580b91a0 2024-04-25T19:11:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465d009640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46580b91a0 0x7f4658163850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465e00b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f46580d2ab0 0x7f465809bae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52346/0 (socket says 172.21.15.28:52346) 2024-04-25T19:11:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465e00b640 1 -- 172.21.15.28:0/2324377305 learned_addr learned my addr 172.21.15.28:0/2324377305 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465d009640 1 -- 172.21.15.28:0/2324377305 >> v1:172.21.15.53:6789/0 conn(0x7f465809cb20 legacy=0x7f46580ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465d009640 1 -- 172.21.15.28:0/2324377305 >> v1:172.21.15.28:6789/0 conn(0x7f46580d2ab0 legacy=0x7f465809bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465c808640 1 -- 172.21.15.28:0/2324377305 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 650754786 0 0) 0x7f46580d1d30 con 0x7f46580d2ab0 2024-04-25T19:11:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.725+0000 7f465d009640 1 -- 172.21.15.28:0/2324377305 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465806e5f0 con 0x7f46580b91a0 2024-04-25T19:11:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.726+0000 7f465d009640 1 --2- 172.21.15.28:0/2324377305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580b91a0 0x7f4658163850 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f465806ed80 tx=0x7f464802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aec60d27de17aae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.726+0000 7f465c808640 1 -- 172.21.15.28:0/2324377305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648037070 con 0x7f46580b91a0 2024-04-25T19:11:41.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.726+0000 7f465c808640 1 -- 172.21.15.28:0/2324377305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4648032440 con 0x7f46580b91a0 2024-04-25T19:11:41.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.726+0000 7f465c808640 1 -- 172.21.15.28:0/2324377305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4648032810 con 0x7f46580b91a0 2024-04-25T19:11:41.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.727+0000 7f465fa6e640 1 -- 172.21.15.28:0/2324377305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580b91a0 msgr2=0x7f4658163850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.727+0000 7f465fa6e640 1 --2- 172.21.15.28:0/2324377305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580b91a0 0x7f4658163850 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f465806ed80 tx=0x7f464802a210 comp rx=0 tx=0).stop 2024-04-25T19:11:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.727+0000 7f465fa6e640 1 -- 172.21.15.28:0/2324377305 shutdown_connections 2024-04-25T19:11:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.727+0000 7f465fa6e640 1 --2- 172.21.15.28:0/2324377305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580b91a0 0x7f4658163850 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.727+0000 7f465fa6e640 1 -- 172.21.15.28:0/2324377305 >> 172.21.15.28:0/2324377305 conn(0x7f46580b54b0 msgr2=0x7f46580b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.727+0000 7f465fa6e640 1 -- 172.21.15.28:0/2324377305 shutdown_connections 2024-04-25T19:11:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.727+0000 7f465fa6e640 1 -- 172.21.15.28:0/2324377305 wait complete. 2024-04-25T19:11:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.729+0000 7f465fa6e640 1 Processor -- start 2024-04-25T19:11:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.729+0000 7f465fa6e640 1 -- start start 2024-04-25T19:11:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465fa6e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f465809cb20 0x7f4658150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465fa6e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580b91a0 0x7f4658152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465fa6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580d2ab0 0x7f46580b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465fa6e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46580d22b0 con 0x7f46580d2ab0 2024-04-25T19:11:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465fa6e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f46580d1fd0 con 0x7f46580b91a0 2024-04-25T19:11:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465fa6e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f46580ce720 con 0x7f465809cb20 2024-04-25T19:11:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465e00b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580d2ab0 0x7f46580b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465e00b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580d2ab0 0x7f46580b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46948/0 (socket says 172.21.15.28:46948) 2024-04-25T19:11:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.730+0000 7f465e00b640 1 -- 172.21.15.28:0/1348519779 learned_addr learned my addr 172.21.15.28:0/1348519779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465d009640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580b91a0 0x7f4658152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465d80a640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f465809cb20 0x7f4658150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465e00b640 1 -- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f465809cb20 msgr2=0x7f4658150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465e00b640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f465809cb20 0x7f4658150900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465e00b640 1 -- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580b91a0 msgr2=0x7f4658152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465e00b640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580b91a0 0x7f4658152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465e00b640 1 -- 172.21.15.28:0/1348519779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4658150e40 con 0x7f46580d2ab0 2024-04-25T19:11:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465d80a640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f465809cb20 0x7f4658150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465e00b640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580d2ab0 0x7f46580b7790 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f465000c6d0 tx=0x7f465000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f464e7fc640 1 -- 172.21.15.28:0/1348519779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650018070 con 0x7f46580d2ab0 2024-04-25T19:11:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f464e7fc640 1 -- 172.21.15.28:0/1348519779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4650004d60 con 0x7f46580d2ab0 2024-04-25T19:11:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46580b66f0 con 0x7f46580d2ab0 2024-04-25T19:11:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f464e7fc640 1 -- 172.21.15.28:0/1348519779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f465000a880 con 0x7f46580d2ab0 2024-04-25T19:11:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.731+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46580b6cb0 con 0x7f46580d2ab0 2024-04-25T19:11:41.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.732+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4658050740 con 0x7f46580d2ab0 2024-04-25T19:11:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.733+0000 7f464e7fc640 1 -- 172.21.15.28:0/1348519779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f465001b070 con 0x7f46580d2ab0 2024-04-25T19:11:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.733+0000 7f464e7fc640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f463c0744b0 0x7f463c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.733+0000 7f464e7fc640 1 -- 172.21.15.28:0/1348519779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4650094ca0 con 0x7f46580d2ab0 2024-04-25T19:11:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.733+0000 7f465d80a640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f463c0744b0 0x7f463c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.734+0000 7f465d80a640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f463c0744b0 0x7f463c0769c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4654000f60 tx=0x7f4654009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:41.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.738+0000 7f464e7fc640 1 -- 172.21.15.28:0/1348519779 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f465005f0f0 con 0x7f46580d2ab0 2024-04-25T19:11:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.924+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46580b2a70 con 0x7f46580d2ab0 2024-04-25T19:11:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.925+0000 7f464e7fc640 1 -- 172.21.15.28:0/1348519779 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f465005e9f0 con 0x7f46580d2ab0 2024-04-25T19:11:41.929 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:41.929 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f463c0744b0 msgr2=0x7f463c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f463c0744b0 0x7f463c0769c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4654000f60 tx=0x7f4654009040 comp rx=0 tx=0).stop 2024-04-25T19:11:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580d2ab0 msgr2=0x7f46580b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580d2ab0 0x7f46580b7790 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f465000c6d0 tx=0x7f465000cb90 comp rx=0 tx=0).stop 2024-04-25T19:11:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 shutdown_connections 2024-04-25T19:11:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f463c0744b0 0x7f463c0769c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46580d2ab0 0x7f46580b7790 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46580b91a0 0x7f4658152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 --2- 172.21.15.28:0/1348519779 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f465809cb20 0x7f4658150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 >> 172.21.15.28:0/1348519779 conn(0x7f46580b54b0 msgr2=0x7f46580714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 shutdown_connections 2024-04-25T19:11:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:41.927+0000 7f465fa6e640 1 -- 172.21.15.28:0/1348519779 wait complete. 2024-04-25T19:11:41.940 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:11:41.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:11:42.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.058+0000 7f57203f3640 1 Processor -- start 2024-04-25T19:11:42.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.058+0000 7f57203f3640 1 -- start start 2024-04-25T19:11:42.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.059+0000 7f57203f3640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f571809caf0 0x7f57180d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.059+0000 7f57203f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180d2670 0x7f57180b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.059+0000 7f57203f3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57180d1ce0 con 0x7f57180b7710 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.059+0000 7f57203f3640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f57180b8fa0 con 0x7f571809caf0 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.059+0000 7f57203f3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f57180cf8d0 con 0x7f57180d2670 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.059+0000 7f571e18f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f571809caf0 0x7f57180d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.059+0000 7f571e18f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f571809caf0 0x7f57180d2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46956/0 (socket says 172.21.15.28:46956) 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.059+0000 7f571e18f640 1 -- 172.21.15.28:0/2427482781 learned_addr learned my addr 172.21.15.28:0/2427482781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.060+0000 7f571d98e640 1 --2- 172.21.15.28:0/2427482781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180d2670 0x7f57180b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.060+0000 7f571e18f640 1 -- 172.21.15.28:0/2427482781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180d2670 msgr2=0x7f57180b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.060+0000 7f571e18f640 1 --2- 172.21.15.28:0/2427482781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180d2670 0x7f57180b71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.060+0000 7f571e18f640 1 -- 172.21.15.28:0/2427482781 >> v1:172.21.15.53:6789/0 conn(0x7f57180b7710 legacy=0x7f5718162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.060+0000 7f571e18f640 1 -- 172.21.15.28:0/2427482781 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57180cfbb0 con 0x7f571809caf0 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.060+0000 7f571d18d640 1 -- 172.21.15.28:0/2427482781 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544711742 0 0) 0x7f57180d1ce0 con 0x7f57180b7710 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.061+0000 7f571e18f640 1 --2- 172.21.15.28:0/2427482781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571809caf0 0x7f57180d2130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5714009870 tx=0x7f571402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86be383d49c5ce63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.061+0000 7f571d18d640 1 -- 172.21.15.28:0/2427482781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714037070 con 0x7f571809caf0 2024-04-25T19:11:42.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.061+0000 7f571d18d640 1 -- 172.21.15.28:0/2427482781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5714032440 con 0x7f571809caf0 2024-04-25T19:11:42.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.061+0000 7f571d18d640 1 -- 172.21.15.28:0/2427482781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714032810 con 0x7f571809caf0 2024-04-25T19:11:42.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.061+0000 7f57203f3640 1 -- 172.21.15.28:0/2427482781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571809caf0 msgr2=0x7f57180d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.061+0000 7f57203f3640 1 --2- 172.21.15.28:0/2427482781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571809caf0 0x7f57180d2130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5714009870 tx=0x7f571402a1c0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.062+0000 7f57203f3640 1 -- 172.21.15.28:0/2427482781 shutdown_connections 2024-04-25T19:11:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.062+0000 7f57203f3640 1 --2- 172.21.15.28:0/2427482781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180d2670 0x7f57180b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.062+0000 7f57203f3640 1 --2- 172.21.15.28:0/2427482781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f571809caf0 0x7f57180d2130 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.062+0000 7f57203f3640 1 -- 172.21.15.28:0/2427482781 >> 172.21.15.28:0/2427482781 conn(0x7f57180d03f0 msgr2=0x7f57180d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.062+0000 7f57203f3640 1 -- 172.21.15.28:0/2427482781 shutdown_connections 2024-04-25T19:11:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.062+0000 7f57203f3640 1 -- 172.21.15.28:0/2427482781 wait complete. 2024-04-25T19:11:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.063+0000 7f57203f3640 1 Processor -- start 2024-04-25T19:11:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.063+0000 7f57203f3640 1 -- start start 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f57203f3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571809caf0 0x7f57181dedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f57203f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b7710 0x7f57181ddb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f57203f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57180d2670 0x7f57180c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f57203f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57180d4790 con 0x7f57180d2670 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f57203f3640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f57180b9430 con 0x7f571809caf0 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f57203f3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f57180d2b10 con 0x7f57180b7710 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57180d2670 0x7f57180c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57180d2670 0x7f57180c91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46970/0 (socket says 172.21.15.28:46970) 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 -- 172.21.15.28:0/2752914646 learned_addr learned my addr 172.21.15.28:0/2752914646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e18f640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571809caf0 0x7f57181dedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571d98e640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b7710 0x7f57181ddb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 -- 172.21.15.28:0/2752914646 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b7710 msgr2=0x7f57181ddb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b7710 0x7f57181ddb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 -- 172.21.15.28:0/2752914646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571809caf0 msgr2=0x7f57181dedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571809caf0 0x7f57181dedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 -- 172.21.15.28:0/2752914646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57181f3b90 con 0x7f57180d2670 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f571e990640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57180d2670 0x7f57180c91b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f571000c780 tx=0x7f571000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.065+0000 7f570effd640 1 -- 172.21.15.28:0/2752914646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5710003da0 con 0x7f57180d2670 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.066+0000 7f570effd640 1 -- 172.21.15.28:0/2752914646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f571000ce30 con 0x7f57180d2670 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.066+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57181df300 con 0x7f57180d2670 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.066+0000 7f570effd640 1 -- 172.21.15.28:0/2752914646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5710005290 con 0x7f57180d2670 2024-04-25T19:11:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.066+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57181dfaf0 con 0x7f57180d2670 2024-04-25T19:11:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.067+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57181df580 con 0x7f57180d2670 2024-04-25T19:11:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.067+0000 7f570effd640 1 -- 172.21.15.28:0/2752914646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5710005430 con 0x7f57180d2670 2024-04-25T19:11:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.068+0000 7f570effd640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f56fc0744b0 0x7f56fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.068+0000 7f570effd640 1 -- 172.21.15.28:0/2752914646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5710095330 con 0x7f57180d2670 2024-04-25T19:11:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.068+0000 7f571e18f640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f56fc0744b0 0x7f56fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.068+0000 7f571e18f640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f56fc0744b0 0x7f56fc0769c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5714034000 tx=0x7f571402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.072+0000 7f570effd640 1 -- 172.21.15.28:0/2752914646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f571005f0b0 con 0x7f57180d2670 2024-04-25T19:11:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.261+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5718050740 con 0x7f57180d2670 2024-04-25T19:11:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.262+0000 7f570effd640 1 -- 172.21.15.28:0/2752914646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f5710080070 con 0x7f57180d2670 2024-04-25T19:11:42.264 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.263+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f56fc0744b0 msgr2=0x7f56fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.263+0000 7f57203f3640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f56fc0744b0 0x7f56fc0769c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5714034000 tx=0x7f571402e000 comp rx=0 tx=0).stop 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57180d2670 msgr2=0x7f57180c91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57180d2670 0x7f57180c91b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f571000c780 tx=0x7f571000cc40 comp rx=0 tx=0).stop 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 shutdown_connections 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f56fc0744b0 0x7f56fc0769c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57180d2670 0x7f57180c91b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f57180b7710 0x7f57181ddb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 --2- 172.21.15.28:0/2752914646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f571809caf0 0x7f57181dedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 >> 172.21.15.28:0/2752914646 conn(0x7f57180d03f0 msgr2=0x7f57180714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 shutdown_connections 2024-04-25T19:11:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.264+0000 7f57203f3640 1 -- 172.21.15.28:0/2752914646 wait complete. 2024-04-25T19:11:42.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:11:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.391+0000 7f7195384640 1 Processor -- start 2024-04-25T19:11:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.391+0000 7f7195384640 1 -- start start 2024-04-25T19:11:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7195384640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f71900b9090 0x7f71900b89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7195384640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71900cf3e0 0x7f71900cea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7195384640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 0x7f71900b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7195384640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f71900d2c10 con 0x7f71900cf3e0 2024-04-25T19:11:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7195384640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f719009d000 con 0x7f71900b9090 2024-04-25T19:11:42.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7195384640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f71900d2650 con 0x7f71900cef40 2024-04-25T19:11:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7187fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71900cf3e0 0x7f71900cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7187fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71900cf3e0 0x7f71900cea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46976/0 (socket says 172.21.15.28:46976) 2024-04-25T19:11:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7187fff640 1 -- 172.21.15.28:0/3903065825 learned_addr learned my addr 172.21.15.28:0/3903065825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f718effd640 1 --2- 172.21.15.28:0/3903065825 >> v2:172.21.15.53:3300/0 conn(0x7f71900b9090 0x7f71900b89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f718f7fe640 1 --2- 172.21.15.28:0/3903065825 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 0x7f71900b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7187fff640 1 -- 172.21.15.28:0/3903065825 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 msgr2=0x7f71900b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7187fff640 1 --2- 172.21.15.28:0/3903065825 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 0x7f71900b3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7187fff640 1 -- 172.21.15.28:0/3903065825 >> v2:172.21.15.53:3300/0 conn(0x7f71900b9090 msgr2=0x7f71900b89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7187fff640 1 --2- 172.21.15.28:0/3903065825 >> v2:172.21.15.53:3300/0 conn(0x7f71900b9090 0x7f71900b89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.392+0000 7f7187fff640 1 -- 172.21.15.28:0/3903065825 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71900d2930 con 0x7f71900cf3e0 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.393+0000 7f7187fff640 1 --2- 172.21.15.28:0/3903065825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900cf3e0 0x7f71900cea00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7190073fc0 tx=0x7f717802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278e26ac2ce1dd97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.393+0000 7f718e7fc640 1 -- 172.21.15.28:0/3903065825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178037070 con 0x7f71900cf3e0 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.393+0000 7f718e7fc640 1 -- 172.21.15.28:0/3903065825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7178032440 con 0x7f71900cf3e0 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.393+0000 7f718e7fc640 1 -- 172.21.15.28:0/3903065825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7178032810 con 0x7f71900cf3e0 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 -- 172.21.15.28:0/3903065825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900cf3e0 msgr2=0x7f71900cea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 --2- 172.21.15.28:0/3903065825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900cf3e0 0x7f71900cea00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7190073fc0 tx=0x7f717802a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 -- 172.21.15.28:0/3903065825 shutdown_connections 2024-04-25T19:11:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 --2- 172.21.15.28:0/3903065825 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 0x7f71900b3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 --2- 172.21.15.28:0/3903065825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900cf3e0 0x7f71900cea00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 --2- 172.21.15.28:0/3903065825 >> v2:172.21.15.53:3300/0 conn(0x7f71900b9090 0x7f71900b89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 -- 172.21.15.28:0/3903065825 >> 172.21.15.28:0/3903065825 conn(0x7f71900d0450 msgr2=0x7f71900d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 -- 172.21.15.28:0/3903065825 shutdown_connections 2024-04-25T19:11:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.394+0000 7f7195384640 1 -- 172.21.15.28:0/3903065825 wait complete. 2024-04-25T19:11:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.396+0000 7f7195384640 1 Processor -- start 2024-04-25T19:11:42.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.396+0000 7f7195384640 1 -- start start 2024-04-25T19:11:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.397+0000 7f7195384640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900b9090 0x7f71901d2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.397+0000 7f7195384640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 0x7f71900ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.397+0000 7f7195384640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71900cf3e0 0x7f71900cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.397+0000 7f7195384640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71900cf7f0 con 0x7f71900b9090 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.397+0000 7f7195384640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f719008cd90 con 0x7f71900cf3e0 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.397+0000 7f7195384640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f71900cf970 con 0x7f71900cef40 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900b9090 0x7f71901d2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900b9090 0x7f71901d2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46980/0 (socket says 172.21.15.28:46980) 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 -- 172.21.15.28:0/912245501 learned_addr learned my addr 172.21.15.28:0/912245501 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f7187fff640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 0x7f71900ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718f7fe640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71900cf3e0 0x7f71900cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 -- 172.21.15.28:0/912245501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 msgr2=0x7f71900ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 0x7f71900ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 -- 172.21.15.28:0/912245501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71900cf3e0 msgr2=0x7f71900cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71900cf3e0 0x7f71900cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 -- 172.21.15.28:0/912245501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f719006f160 con 0x7f71900b9090 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f718effd640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900b9090 0x7f71901d2b90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f718000d800 tx=0x7f718000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f71877fe640 1 -- 172.21.15.28:0/912245501 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180018030 con 0x7f71900b9090 2024-04-25T19:11:42.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f71877fe640 1 -- 172.21.15.28:0/912245501 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7180009640 con 0x7f71900b9090 2024-04-25T19:11:42.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f719006f440 con 0x7f71900b9090 2024-04-25T19:11:42.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f71877fe640 1 -- 172.21.15.28:0/912245501 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7180009810 con 0x7f71900b9090 2024-04-25T19:11:42.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.398+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f719006e090 con 0x7f71900b9090 2024-04-25T19:11:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.399+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f719006e380 con 0x7f71900b9090 2024-04-25T19:11:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.400+0000 7f71877fe640 1 -- 172.21.15.28:0/912245501 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f718001b070 con 0x7f71900b9090 2024-04-25T19:11:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.401+0000 7f71877fe640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71740744b0 0x7f71740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.401+0000 7f7187fff640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71740744b0 0x7f71740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.401+0000 7f71877fe640 1 -- 172.21.15.28:0/912245501 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7180095520 con 0x7f71900b9090 2024-04-25T19:11:42.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.401+0000 7f7187fff640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71740744b0 0x7f71740769c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7178034000 tx=0x7f717802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.405+0000 7f71877fe640 1 -- 172.21.15.28:0/912245501 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f718005f3a0 con 0x7f71900b9090 2024-04-25T19:11:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.590+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7190050740 con 0x7f71900b9090 2024-04-25T19:11:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.590+0000 7f71877fe640 1 -- 172.21.15.28:0/912245501 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f71800636a0 con 0x7f71900b9090 2024-04-25T19:11:42.592 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:11:42.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71740744b0 msgr2=0x7f71740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71740744b0 0x7f71740769c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7178034000 tx=0x7f717802e000 comp rx=0 tx=0).stop 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900b9090 msgr2=0x7f71901d2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900b9090 0x7f71901d2b90 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f718000d800 tx=0x7f718000dcc0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 shutdown_connections 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71740744b0 0x7f71740769c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71900cf3e0 0x7f71900cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71900cef40 0x7f71900ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.592+0000 7f7195384640 1 --2- 172.21.15.28:0/912245501 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71900b9090 0x7f71901d2b90 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.593+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 >> 172.21.15.28:0/912245501 conn(0x7f71900d0450 msgr2=0x7f71900b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.593+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 shutdown_connections 2024-04-25T19:11:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.593+0000 7f7195384640 1 -- 172.21.15.28:0/912245501 wait complete. 2024-04-25T19:11:42.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:11:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.720+0000 7fc01acf0640 1 Processor -- start 2024-04-25T19:11:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.720+0000 7fc01acf0640 1 -- start start 2024-04-25T19:11:42.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.721+0000 7fc01acf0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc014160250 0x7fc0140b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.721+0000 7fc01acf0640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc014164b20 con 0x7fc0140d2ca0 2024-04-25T19:11:42.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.721+0000 7fc01acf0640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc014163d60 con 0x7fc0140b81a0 2024-04-25T19:11:42.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.721+0000 7fc01acf0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc0140b4f50 con 0x7fc014160250 2024-04-25T19:11:42.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.721+0000 7fc01928d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc0140d2ca0 0x7fc0140d44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52348/0 (socket says 172.21.15.28:52348) 2024-04-25T19:11:42.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.721+0000 7fc01928d640 1 -- 172.21.15.28:0/1506906504 learned_addr learned my addr 172.21.15.28:0/1506906504 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:42.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc018a8c640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc014160250 0x7fc0140b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1324065682 0 0) 0x7fc014164b20 con 0x7fc0140d2ca0 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbffc003520 con 0x7fc0140d2ca0 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3193268081 0 0) 0x7fc014163d60 con 0x7fc0140b81a0 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc014164b20 con 0x7fc0140b81a0 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 617903750 0 0) 0x7fbffc003520 con 0x7fc0140d2ca0 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc014160250 msgr2=0x7fc0140b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc014160250 0x7fc0140b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 >> v1:172.21.15.53:6789/0 conn(0x7fc0140b81a0 legacy=0x7fc0140d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc014164040 con 0x7fc0140d2ca0 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fc00c0019f0 con 0x7fc0140d2ca0 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 >> v1:172.21.15.28:6789/0 conn(0x7fc0140d2ca0 legacy=0x7fc0140d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbffc003b70 0x7fbffc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc014160250 0x7fbffc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbffc004d40 0x7fbffc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0140b4f50 con 0x7fbffc004d40 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc00bbd0 con 0x7fbffc003b70 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbffc00bdd0 con 0x7fc014160250 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc01928d640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbffc004d40 0x7fbffc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc018a8c640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbffc003b70 0x7fbffc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.722+0000 7fc00bfff640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc014160250 0x7fbffc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc01928d640 1 -- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc014160250 msgr2=0x7fbffc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc01928d640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc014160250 0x7fbffc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc01928d640 1 -- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbffc003b70 msgr2=0x7fbffc003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc01928d640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbffc003b70 0x7fbffc003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc01928d640 1 -- 172.21.15.28:0/1506906504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc003520 con 0x7fbffc004d40 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc01928d640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbffc004d40 0x7fbffc005190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc00c002230 tx=0x7fbffc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b57eabbc4bba56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c00a370 con 0x7fbffc004d40 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00c02f070 con 0x7fbffc004d40 2024-04-25T19:11:42.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.723+0000 7fc00b7fe640 1 -- 172.21.15.28:0/1506906504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00c02bb80 con 0x7fbffc004d40 2024-04-25T19:11:42.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.724+0000 7fc01acf0640 1 -- 172.21.15.28:0/1506906504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbffc004d40 msgr2=0x7fbffc005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.724+0000 7fc01acf0640 1 --2- 172.21.15.28:0/1506906504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbffc004d40 0x7fbffc005190 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc00c002230 tx=0x7fbffc005830 comp rx=0 tx=0).stop 2024-04-25T19:11:42.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.724+0000 7fc01928d640 1 -- 172.21.15.28:0/1506906504 reap_dead start 2024-04-25T19:11:42.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.724+0000 7fc01acf0640 1 -- 172.21.15.28:0/1506906504 shutdown_connections 2024-04-25T19:11:42.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.724+0000 7fc01acf0640 1 -- 172.21.15.28:0/1506906504 >> 172.21.15.28:0/1506906504 conn(0x7fc01406ee30 msgr2=0x7fc01406c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:42.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.724+0000 7fc01acf0640 1 -- 172.21.15.28:0/1506906504 shutdown_connections 2024-04-25T19:11:42.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.724+0000 7fc01acf0640 1 -- 172.21.15.28:0/1506906504 wait complete. 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.726+0000 7fc01acf0640 1 Processor -- start 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.726+0000 7fc01acf0640 1 -- start start 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc01acf0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbffc004d40 0x7fc0140c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc01acf0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc014159670 0x7fc014156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc01acf0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014157370 0x7fc014147750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc01acf0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc0140b4690 con 0x7fc014159670 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc01acf0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc014164600 con 0x7fc014157370 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc01acf0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc0140b4810 con 0x7fbffc004d40 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc00bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc014159670 0x7fc014156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc00bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc014159670 0x7fc014156e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46992/0 (socket says 172.21.15.28:46992) 2024-04-25T19:11:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.727+0000 7fc00bfff640 1 -- 172.21.15.28:0/297708450 learned_addr learned my addr 172.21.15.28:0/297708450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc01928d640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014157370 0x7fc014147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc018a8c640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbffc004d40 0x7fc0140c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc00bfff640 1 -- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbffc004d40 msgr2=0x7fc0140c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc00bfff640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbffc004d40 0x7fc0140c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc00bfff640 1 -- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014157370 msgr2=0x7fc014147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc00bfff640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014157370 0x7fc014147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc00bfff640 1 -- 172.21.15.28:0/297708450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0141585d0 con 0x7fc014159670 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc018a8c640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbffc004d40 0x7fc0140c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc01928d640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014157370 0x7fc014147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc00bfff640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc014159670 0x7fc014156e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc00400f7a0 tx=0x7fc00400fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc0097fa640 1 -- 172.21.15.28:0/297708450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc00401a070 con 0x7fc014159670 2024-04-25T19:11:42.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc0097fa640 1 -- 172.21.15.28:0/297708450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00400fe50 con 0x7fc014159670 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0141588b0 con 0x7fc014159670 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc0097fa640 1 -- 172.21.15.28:0/297708450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc004007050 con 0x7fc014159670 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.728+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc014158e70 con 0x7fc014159670 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.729+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc014050740 con 0x7fc014159670 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.730+0000 7fc0097fa640 1 -- 172.21.15.28:0/297708450 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc00401d070 con 0x7fc014159670 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.730+0000 7fc0097fa640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc0000744b0 0x7fc0000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.731+0000 7fc018a8c640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc0000744b0 0x7fc0000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.731+0000 7fc0097fa640 1 -- 172.21.15.28:0/297708450 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc004094df0 con 0x7fc014159670 2024-04-25T19:11:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.731+0000 7fc018a8c640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc0000744b0 0x7fc0000769c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc0140d05f0 tx=0x7fc010008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.735+0000 7fc0097fa640 1 -- 172.21.15.28:0/297708450 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc00405f2d0 con 0x7fc014159670 2024-04-25T19:11:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.921+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc0140d3fd0 con 0x7fc014159670 2024-04-25T19:11:42.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.921+0000 7fc0097fa640 1 -- 172.21.15.28:0/297708450 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7fc00405ebd0 con 0x7fc014159670 2024-04-25T19:11:42.924 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.923+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc0000744b0 msgr2=0x7fc0000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.923+0000 7fc01acf0640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc0000744b0 0x7fc0000769c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc0140d05f0 tx=0x7fc010008040 comp rx=0 tx=0).stop 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc014159670 msgr2=0x7fc014156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc014159670 0x7fc014156e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc00400f7a0 tx=0x7fc00400fc60 comp rx=0 tx=0).stop 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 shutdown_connections 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc0000744b0 0x7fc0000769c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc014157370 0x7fc014147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc014159670 0x7fc014156e30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 --2- 172.21.15.28:0/297708450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbffc004d40 0x7fc0140c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:42.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 >> 172.21.15.28:0/297708450 conn(0x7fc01406ee30 msgr2=0x7fc0140b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:42.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 shutdown_connections 2024-04-25T19:11:42.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:42.924+0000 7fc01acf0640 1 -- 172.21.15.28:0/297708450 wait complete. 2024-04-25T19:11:42.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:11:43.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.052+0000 7fb8890e7640 1 Processor -- start 2024-04-25T19:11:43.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.052+0000 7fb8890e7640 1 -- start start 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb8890e7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8840b9000 0x7fb8840d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb8890e7640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb88406e4a0 0x7fb88406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb8890e7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8840ce980 0x7fb8840ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb8890e7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb8840d0ec0 con 0x7fb8840b9000 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb8890e7640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fb88409cd00 con 0x7fb88406e4a0 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb8890e7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb8840b9830 con 0x7fb8840ce980 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb882d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8840b9000 0x7fb8840d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb882d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8840b9000 0x7fb8840d3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47002/0 (socket says 172.21.15.28:47002) 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.053+0000 7fb882d9d640 1 -- 172.21.15.28:0/3463012655 learned_addr learned my addr 172.21.15.28:0/3463012655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.054+0000 7fb88259c640 1 --2- 172.21.15.28:0/3463012655 >> v2:172.21.15.53:3300/0 conn(0x7fb88406e4a0 0x7fb88406e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.054+0000 7fb88359e640 1 --2- 172.21.15.28:0/3463012655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8840ce980 0x7fb8840ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.054+0000 7fb882d9d640 1 -- 172.21.15.28:0/3463012655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8840ce980 msgr2=0x7fb8840ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.054+0000 7fb882d9d640 1 --2- 172.21.15.28:0/3463012655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8840ce980 0x7fb8840ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.054+0000 7fb882d9d640 1 -- 172.21.15.28:0/3463012655 >> v2:172.21.15.53:3300/0 conn(0x7fb88406e4a0 msgr2=0x7fb88406e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.054+0000 7fb882d9d640 1 --2- 172.21.15.28:0/3463012655 >> v2:172.21.15.53:3300/0 conn(0x7fb88406e4a0 0x7fb88406e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.054+0000 7fb882d9d640 1 -- 172.21.15.28:0/3463012655 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88409d230 con 0x7fb8840b9000 2024-04-25T19:11:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.054+0000 7fb882d9d640 1 --2- 172.21.15.28:0/3463012655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840b9000 0x7fb8840d3310 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb88409d400 tx=0x7fb87402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccce36bdb9a22dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb881d9b640 1 -- 172.21.15.28:0/3463012655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874036070 con 0x7fb8840b9000 2024-04-25T19:11:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb881d9b640 1 -- 172.21.15.28:0/3463012655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb874031600 con 0x7fb8840b9000 2024-04-25T19:11:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb881d9b640 1 -- 172.21.15.28:0/3463012655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8740319d0 con 0x7fb8840b9000 2024-04-25T19:11:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 -- 172.21.15.28:0/3463012655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840b9000 msgr2=0x7fb8840d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 --2- 172.21.15.28:0/3463012655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840b9000 0x7fb8840d3310 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb88409d400 tx=0x7fb87402a680 comp rx=0 tx=0).stop 2024-04-25T19:11:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 -- 172.21.15.28:0/3463012655 shutdown_connections 2024-04-25T19:11:43.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 --2- 172.21.15.28:0/3463012655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb8840ce980 0x7fb8840ced90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 --2- 172.21.15.28:0/3463012655 >> v2:172.21.15.53:3300/0 conn(0x7fb88406e4a0 0x7fb88406e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 --2- 172.21.15.28:0/3463012655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840b9000 0x7fb8840d3310 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 -- 172.21.15.28:0/3463012655 >> 172.21.15.28:0/3463012655 conn(0x7fb8840d03a0 msgr2=0x7fb8840d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 -- 172.21.15.28:0/3463012655 shutdown_connections 2024-04-25T19:11:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.055+0000 7fb8890e7640 1 -- 172.21.15.28:0/3463012655 wait complete. 2024-04-25T19:11:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.057+0000 7fb8890e7640 1 Processor -- start 2024-04-25T19:11:43.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.057+0000 7fb8890e7640 1 -- start start 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb8890e7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb88406e4a0 0x7fb88415a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb8890e7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8840b9000 0x7fb8841f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb8890e7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840ce980 0x7fb8841f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb8890e7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb88406d680 con 0x7fb8840ce980 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb8890e7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb88409cf50 con 0x7fb8840b9000 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb8890e7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb88406d800 con 0x7fb88406e4a0 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840ce980 0x7fb8841f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840ce980 0x7fb8841f61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47014/0 (socket says 172.21.15.28:47014) 2024-04-25T19:11:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 -- 172.21.15.28:0/1566438462 learned_addr learned my addr 172.21.15.28:0/1566438462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb882d9d640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb88406e4a0 0x7fb88415a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88259c640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8840b9000 0x7fb8841f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 -- 172.21.15.28:0/1566438462 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb88406e4a0 msgr2=0x7fb88415a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb88406e4a0 0x7fb88415a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 -- 172.21.15.28:0/1566438462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8840b9000 msgr2=0x7fb8841f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8840b9000 0x7fb8841f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 -- 172.21.15.28:0/1566438462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb88415ab80 con 0x7fb8840ce980 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.059+0000 7fb88359e640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840ce980 0x7fb8841f61a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00d600 tx=0x7fb87c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.060+0000 7fb87b7fe640 1 -- 172.21.15.28:0/1566438462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0125f0 con 0x7fb8840ce980 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.060+0000 7fb87b7fe640 1 -- 172.21.15.28:0/1566438462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb87c019070 con 0x7fb8840ce980 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.060+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb88415d400 con 0x7fb8840ce980 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.060+0000 7fb87b7fe640 1 -- 172.21.15.28:0/1566438462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb87c0129f0 con 0x7fb8840ce980 2024-04-25T19:11:43.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.060+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb88415d9c0 con 0x7fb8840ce980 2024-04-25T19:11:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.061+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb884050740 con 0x7fb8840ce980 2024-04-25T19:11:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.061+0000 7fb87b7fe640 1 -- 172.21.15.28:0/1566438462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb87c012b90 con 0x7fb8840ce980 2024-04-25T19:11:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.062+0000 7fb87b7fe640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb8640744e0 0x7fb8640769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.062+0000 7fb87b7fe640 1 -- 172.21.15.28:0/1566438462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb87c0957b0 con 0x7fb8840ce980 2024-04-25T19:11:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.062+0000 7fb882d9d640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb8640744e0 0x7fb8640769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.062+0000 7fb882d9d640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb8640744e0 0x7fb8640769f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb874033000 tx=0x7fb874009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.066+0000 7fb87b7fe640 1 -- 172.21.15.28:0/1566438462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb87c05f520 con 0x7fb8840ce980 2024-04-25T19:11:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.253+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb88415ae20 con 0x7fb8840ce980 2024-04-25T19:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.253+0000 7fb87b7fe640 1 -- 172.21.15.28:0/1566438462 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fb87c063820 con 0x7fb8840ce980 2024-04-25T19:11:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:11:43.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb8640744e0 msgr2=0x7fb8640769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb8640744e0 0x7fb8640769f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb874033000 tx=0x7fb874009000 comp rx=0 tx=0).stop 2024-04-25T19:11:43.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840ce980 msgr2=0x7fb8841f61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840ce980 0x7fb8841f61a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00d600 tx=0x7fb87c00dac0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 shutdown_connections 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb8640744e0 0x7fb8640769f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8840ce980 0x7fb8841f61a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8840b9000 0x7fb8841f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 --2- 172.21.15.28:0/1566438462 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb88406e4a0 0x7fb88415a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 >> 172.21.15.28:0/1566438462 conn(0x7fb8840d03a0 msgr2=0x7fb884072750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 shutdown_connections 2024-04-25T19:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.255+0000 7fb8890e7640 1 -- 172.21.15.28:0/1566438462 wait complete. 2024-04-25T19:11:43.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:11:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.378+0000 7fcaf7584640 1 Processor -- start 2024-04-25T19:11:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.379+0000 7fcaf7584640 1 -- start start 2024-04-25T19:11:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf7584640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcaf0160250 0x7fcaf00b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf7584640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcaf00d2ca0 0x7fcaf00d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf7584640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaf0164b20 con 0x7fcaf00b81a0 2024-04-25T19:11:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf7584640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fcaf00b4f50 con 0x7fcaf0160250 2024-04-25T19:11:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf7584640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fcaf0163d60 con 0x7fcaf00d2ca0 2024-04-25T19:11:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf5320640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcaf0160250 0x7fcaf00b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf5320640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcaf0160250 0x7fcaf00b4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47024/0 (socket says 172.21.15.28:47024) 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf5320640 1 -- 172.21.15.28:0/3513578825 learned_addr learned my addr 172.21.15.28:0/3513578825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf5b21640 1 --2- 172.21.15.28:0/3513578825 >> v2:172.21.15.53:3300/0 conn(0x7fcaf00d2ca0 0x7fcaf00d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf5320640 1 -- 172.21.15.28:0/3513578825 >> v2:172.21.15.53:3300/0 conn(0x7fcaf00d2ca0 msgr2=0x7fcaf00d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf5320640 1 --2- 172.21.15.28:0/3513578825 >> v2:172.21.15.53:3300/0 conn(0x7fcaf00d2ca0 0x7fcaf00d44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf5320640 1 -- 172.21.15.28:0/3513578825 >> v1:172.21.15.53:6789/0 conn(0x7fcaf00b81a0 legacy=0x7fcaf00d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.380+0000 7fcaf5320640 1 -- 172.21.15.28:0/3513578825 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf0164040 con 0x7fcaf0160250 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.381+0000 7fcaf5320640 1 --2- 172.21.15.28:0/3513578825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0160250 0x7fcaf00b4690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcaf00d3890 tx=0x7fcae802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c9f35eb5e9ed03d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.381+0000 7fcaeffff640 1 -- 172.21.15.28:0/3513578825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8037070 con 0x7fcaf0160250 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.381+0000 7fcaeffff640 1 -- 172.21.15.28:0/3513578825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae8032440 con 0x7fcaf0160250 2024-04-25T19:11:43.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.381+0000 7fcaeffff640 1 -- 172.21.15.28:0/3513578825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8032810 con 0x7fcaf0160250 2024-04-25T19:11:43.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.382+0000 7fcaf7584640 1 -- 172.21.15.28:0/3513578825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0160250 msgr2=0x7fcaf00b4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.382+0000 7fcaf7584640 1 --2- 172.21.15.28:0/3513578825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0160250 0x7fcaf00b4690 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcaf00d3890 tx=0x7fcae802a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.382+0000 7fcaf7584640 1 -- 172.21.15.28:0/3513578825 shutdown_connections 2024-04-25T19:11:43.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.382+0000 7fcaf7584640 1 --2- 172.21.15.28:0/3513578825 >> v2:172.21.15.53:3300/0 conn(0x7fcaf00d2ca0 0x7fcaf00d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.382+0000 7fcaf7584640 1 --2- 172.21.15.28:0/3513578825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf0160250 0x7fcaf00b4690 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.382+0000 7fcaf7584640 1 -- 172.21.15.28:0/3513578825 >> 172.21.15.28:0/3513578825 conn(0x7fcaf006ee30 msgr2=0x7fcaf006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:43.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.382+0000 7fcaf7584640 1 -- 172.21.15.28:0/3513578825 shutdown_connections 2024-04-25T19:11:43.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.382+0000 7fcaf7584640 1 -- 172.21.15.28:0/3513578825 wait complete. 2024-04-25T19:11:43.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.384+0000 7fcaf7584640 1 Processor -- start 2024-04-25T19:11:43.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.384+0000 7fcaf7584640 1 -- start start 2024-04-25T19:11:43.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf7584640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b81a0 0x7fcaf014bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf7584640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0160250 0x7fcaf01620d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf7584640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf01eb0c0 0x7fcaf01eb4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf7584640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf00b4bd0 con 0x7fcaf00b81a0 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf7584640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf0164600 con 0x7fcaf0160250 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf7584640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf00d49f0 con 0x7fcaf01eb0c0 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf5320640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b81a0 0x7fcaf014bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf5320640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b81a0 0x7fcaf014bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47032/0 (socket says 172.21.15.28:47032) 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.385+0000 7fcaf5320640 1 -- 172.21.15.28:0/1691554485 learned_addr learned my addr 172.21.15.28:0/1691554485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf4b1f640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0160250 0x7fcaf01620d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf5320640 1 -- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf01eb0c0 msgr2=0x7fcaf01eb4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf5b21640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf01eb0c0 0x7fcaf01eb4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf5320640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf01eb0c0 0x7fcaf01eb4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf5320640 1 -- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0160250 msgr2=0x7fcaf01620d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf5320640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0160250 0x7fcaf01620d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf5320640 1 -- 172.21.15.28:0/1691554485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf01eba10 con 0x7fcaf00b81a0 2024-04-25T19:11:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf4b1f640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0160250 0x7fcaf01620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf5b21640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf01eb0c0 0x7fcaf01eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf5320640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b81a0 0x7fcaf014bef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcae8000c00 tx=0x7fcae80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaedffb640 1 -- 172.21.15.28:0/1691554485 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8041020 con 0x7fcaf00b81a0 2024-04-25T19:11:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaedffb640 1 -- 172.21.15.28:0/1691554485 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae80306c0 con 0x7fcaf00b81a0 2024-04-25T19:11:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaedffb640 1 -- 172.21.15.28:0/1691554485 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8037040 con 0x7fcaf00b81a0 2024-04-25T19:11:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf0162b90 con 0x7fcaf00b81a0 2024-04-25T19:11:43.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.386+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf0163150 con 0x7fcaf00b81a0 2024-04-25T19:11:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.387+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf0050740 con 0x7fcaf00b81a0 2024-04-25T19:11:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.388+0000 7fcaedffb640 1 -- 172.21.15.28:0/1691554485 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcae8044020 con 0x7fcaf00b81a0 2024-04-25T19:11:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.389+0000 7fcaedffb640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcadc0744e0 0x7fcadc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.389+0000 7fcaedffb640 1 -- 172.21.15.28:0/1691554485 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcae80b5da0 con 0x7fcaf00b81a0 2024-04-25T19:11:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.389+0000 7fcaf4b1f640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcadc0744e0 0x7fcadc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:43.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.389+0000 7fcaf4b1f640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcadc0744e0 0x7fcadc0769f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcaf01ebe30 tx=0x7fcae4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.393+0000 7fcaedffb640 1 -- 172.21.15.28:0/1691554485 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcae804b030 con 0x7fcaf00b81a0 2024-04-25T19:11:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.592+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcaf00d3fd0 con 0x7fcaf00b81a0 2024-04-25T19:11:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.592+0000 7fcaedffb640 1 -- 172.21.15.28:0/1691554485 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7fcaf01eba10 con 0x7fcaf00b81a0 2024-04-25T19:11:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-04-25T19:11:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcadc0744e0 msgr2=0x7fcadc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcadc0744e0 0x7fcadc0769f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcaf01ebe30 tx=0x7fcae4009040 comp rx=0 tx=0).stop 2024-04-25T19:11:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b81a0 msgr2=0x7fcaf014bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b81a0 0x7fcaf014bef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcae8000c00 tx=0x7fcae80046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 shutdown_connections 2024-04-25T19:11:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcadc0744e0 0x7fcadc0769f0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcaf01ebe30 tx=0x7fcae4009040 comp rx=0 tx=0).stop 2024-04-25T19:11:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf01eb0c0 0x7fcaf01eb4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf0160250 0x7fcaf01620d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 --2- 172.21.15.28:0/1691554485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b81a0 0x7fcaf014bef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 >> 172.21.15.28:0/1691554485 conn(0x7fcaf006ee30 msgr2=0x7fcaf00b6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 shutdown_connections 2024-04-25T19:11:43.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:43.594+0000 7fcaf7584640 1 -- 172.21.15.28:0/1691554485 wait complete. 2024-04-25T19:11:43.607 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi028.front.sepia.ceph.com... 2024-04-25T19:11:43.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:43.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T19:11:43.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi028.stderr:ceph-fuse[41467]: fuse finished with error 0 and tester_r 0 2024-04-25T19:11:43.632 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:11:49.733 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:11:49.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:11:49.734 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T19:11:49.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.929+0000 7fcc1b896640 1 Processor -- start 2024-04-25T19:11:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.929+0000 7fcc1b896640 1 -- start start 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.930+0000 7fcc1b896640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc140b5130 0x7fcc14161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.930+0000 7fcc1b896640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc14163e80 con 0x7fcc1415f7d0 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.930+0000 7fcc1b896640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc14164160 con 0x7fcc1406ded0 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.930+0000 7fcc1b896640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fcc14163c30 con 0x7fcc140b5130 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.931+0000 7fcc19e33640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcc1415f7d0 0x7fcc1415fbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52362/0 (socket says 172.21.15.28:52362) 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.931+0000 7fcc19e33640 1 -- 172.21.15.28:0/2435883138 learned_addr learned my addr 172.21.15.28:0/2435883138 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.931+0000 7fcc19632640 1 --2- 172.21.15.28:0/2435883138 >> v2:172.21.15.53:3300/0 conn(0x7fcc140b5130 0x7fcc14161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.931+0000 7fcc0bfff640 1 -- 172.21.15.28:0/2435883138 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171638200 0 0) 0x7fcc14163e80 con 0x7fcc1415f7d0 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.931+0000 7fcc0bfff640 1 -- 172.21.15.28:0/2435883138 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc00003520 con 0x7fcc1415f7d0 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.931+0000 7fcc19632640 1 -- 172.21.15.28:0/2435883138 >> v1:172.21.15.53:6789/0 conn(0x7fcc1406ded0 legacy=0x7fcc1406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.931+0000 7fcc19632640 1 -- 172.21.15.28:0/2435883138 >> v1:172.21.15.28:6789/0 conn(0x7fcc1415f7d0 legacy=0x7fcc1415fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.931+0000 7fcc19632640 1 -- 172.21.15.28:0/2435883138 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc14164440 con 0x7fcc140b5130 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.932+0000 7fcc19632640 1 --2- 172.21.15.28:0/2435883138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc140b5130 0x7fcc14161130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcc100097a0 tx=0x7fcc1002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b66d8a24c978d4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.932+0000 7fcc0bfff640 1 -- 172.21.15.28:0/2435883138 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc10037070 con 0x7fcc140b5130 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.932+0000 7fcc0bfff640 1 -- 172.21.15.28:0/2435883138 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc10032440 con 0x7fcc140b5130 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.932+0000 7fcc0bfff640 1 -- 172.21.15.28:0/2435883138 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc100327f0 con 0x7fcc140b5130 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.933+0000 7fcc1b896640 1 -- 172.21.15.28:0/2435883138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc140b5130 msgr2=0x7fcc14161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.933+0000 7fcc1b896640 1 --2- 172.21.15.28:0/2435883138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc140b5130 0x7fcc14161130 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcc100097a0 tx=0x7fcc1002a230 comp rx=0 tx=0).stop 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.933+0000 7fcc1b896640 1 -- 172.21.15.28:0/2435883138 shutdown_connections 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.933+0000 7fcc1b896640 1 --2- 172.21.15.28:0/2435883138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc140b5130 0x7fcc14161130 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.933+0000 7fcc1b896640 1 -- 172.21.15.28:0/2435883138 >> 172.21.15.28:0/2435883138 conn(0x7fcc140b7ad0 msgr2=0x7fcc140b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.933+0000 7fcc1b896640 1 -- 172.21.15.28:0/2435883138 shutdown_connections 2024-04-25T19:11:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.933+0000 7fcc1b896640 1 -- 172.21.15.28:0/2435883138 wait complete. 2024-04-25T19:11:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.935+0000 7fcc1b896640 1 Processor -- start 2024-04-25T19:11:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.935+0000 7fcc1b896640 1 -- start start 2024-04-25T19:11:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc1b896640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1406ded0 0x7fcc14157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc1b896640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc140b5130 0x7fcc1415aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc1b896640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1415f7d0 0x7fcc141f6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc1b896640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcc140d4910 con 0x7fcc1415f7d0 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc1b896640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcc14164a90 con 0x7fcc1406ded0 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc1b896640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcc140d4a90 con 0x7fcc140b5130 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc19e33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1415f7d0 0x7fcc141f6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc19e33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1415f7d0 0x7fcc141f6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47038/0 (socket says 172.21.15.28:47038) 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc19e33640 1 -- 172.21.15.28:0/61635646 learned_addr learned my addr 172.21.15.28:0/61635646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc19632640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1406ded0 0x7fcc14157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.936+0000 7fcc18e31640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc140b5130 0x7fcc1415aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc19e33640 1 -- 172.21.15.28:0/61635646 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc140b5130 msgr2=0x7fcc1415aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc19e33640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc140b5130 0x7fcc1415aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc19e33640 1 -- 172.21.15.28:0/61635646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1406ded0 msgr2=0x7fcc14157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc19e33640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1406ded0 0x7fcc14157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc19e33640 1 -- 172.21.15.28:0/61635646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc141f72c0 con 0x7fcc1415f7d0 2024-04-25T19:11:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc19e33640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1415f7d0 0x7fcc141f6d80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00cad0 tx=0x7fcc0c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc09ffb640 1 -- 172.21.15.28:0/61635646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c017070 con 0x7fcc1415f7d0 2024-04-25T19:11:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc09ffb640 1 -- 172.21.15.28:0/61635646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc0c00ac30 con 0x7fcc1415f7d0 2024-04-25T19:11:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc14159c00 con 0x7fcc1415f7d0 2024-04-25T19:11:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc09ffb640 1 -- 172.21.15.28:0/61635646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c012400 con 0x7fcc1415f7d0 2024-04-25T19:11:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.937+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc1415a1c0 con 0x7fcc1415f7d0 2024-04-25T19:11:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.938+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc14050740 con 0x7fcc1415f7d0 2024-04-25T19:11:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.939+0000 7fcc09ffb640 1 -- 172.21.15.28:0/61635646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcc0c01a070 con 0x7fcc1415f7d0 2024-04-25T19:11:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.939+0000 7fcc09ffb640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcbf80744e0 0x7fcbf80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.940+0000 7fcc19632640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcbf80744e0 0x7fcbf80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.940+0000 7fcc09ffb640 1 -- 172.21.15.28:0/61635646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcc0c095250 con 0x7fcc1415f7d0 2024-04-25T19:11:49.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.940+0000 7fcc19632640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcbf80744e0 0x7fcbf80769f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcc10034000 tx=0x7fcc1002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:49.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:49.944+0000 7fcc09ffb640 1 -- 172.21.15.28:0/61635646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcc141f72c0 con 0x7fcc1415f7d0 2024-04-25T19:11:50.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.134+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcc140d2fe0 con 0x7fcc1415f7d0 2024-04-25T19:11:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.135+0000 7fcc09ffb640 1 -- 172.21.15.28:0/61635646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7fcc0c05f050 con 0x7fcc1415f7d0 2024-04-25T19:11:50.137 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:50.137 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcbf80744e0 msgr2=0x7fcbf80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcbf80744e0 0x7fcbf80769f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcc10034000 tx=0x7fcc1002e000 comp rx=0 tx=0).stop 2024-04-25T19:11:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1415f7d0 msgr2=0x7fcc141f6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1415f7d0 0x7fcc141f6d80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c00cad0 tx=0x7fcc0c00cf90 comp rx=0 tx=0).stop 2024-04-25T19:11:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 shutdown_connections 2024-04-25T19:11:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcbf80744e0 0x7fcbf80769f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1415f7d0 0x7fcc141f6d80 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcc140b5130 0x7fcc1415aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 --2- 172.21.15.28:0/61635646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc1406ded0 0x7fcc14157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 >> 172.21.15.28:0/61635646 conn(0x7fcc140b7ad0 msgr2=0x7fcc140d5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 shutdown_connections 2024-04-25T19:11:50.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.137+0000 7fcc1b896640 1 -- 172.21.15.28:0/61635646 wait complete. 2024-04-25T19:11:50.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.261+0000 7fd07835e640 1 Processor -- start 2024-04-25T19:11:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.261+0000 7fd07835e640 1 -- start start 2024-04-25T19:11:50.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.262+0000 7fd07835e640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0700b8050 0x7fd070160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.262+0000 7fd07835e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0700d4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.262+0000 7fd07835e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd070160530 con 0x7fd070160120 2024-04-25T19:11:50.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.262+0000 7fd07835e640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fd070164580 con 0x7fd0700b8050 2024-04-25T19:11:50.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.262+0000 7fd07835e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd0701614f0 con 0x7fd070160cc0 2024-04-25T19:11:50.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0768fb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0700d4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0758f9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0700b8050 0x7fd070160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0768fb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0700d4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:50620/0 (socket says 172.21.15.28:50620) 2024-04-25T19:11:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0768fb640 1 -- 172.21.15.28:0/2184448536 learned_addr learned my addr 172.21.15.28:0/2184448536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0760fa640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd070160120 0x7fd0700b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:42542/0 (socket says 172.21.15.28:42542) 2024-04-25T19:11:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0758f9640 1 -- 172.21.15.28:0/2184448536 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 msgr2=0x7fd0700d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0758f9640 1 --2- 172.21.15.28:0/2184448536 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0700d4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0758f9640 1 -- 172.21.15.28:0/2184448536 >> v1:172.21.15.53:6789/0 conn(0x7fd070160120 legacy=0x7fd0700b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0750f8640 1 -- 172.21.15.28:0/2184448536 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 192976097 0 0) 0x7fd06c001110 con 0x7fd070160120 2024-04-25T19:11:50.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.263+0000 7fd0758f9640 1 -- 172.21.15.28:0/2184448536 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd070163c50 con 0x7fd0700b8050 2024-04-25T19:11:50.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.264+0000 7fd0758f9640 1 --2- 172.21.15.28:0/2184448536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0700b8050 0x7fd070160780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd0600097a0 tx=0x7fd06002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0d2f4a91030fd03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:50.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.264+0000 7fd0750f8640 1 -- 172.21.15.28:0/2184448536 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060037070 con 0x7fd0700b8050 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.264+0000 7fd0750f8640 1 -- 172.21.15.28:0/2184448536 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd060032440 con 0x7fd0700b8050 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.264+0000 7fd0750f8640 1 -- 172.21.15.28:0/2184448536 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd060032810 con 0x7fd0700b8050 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.265+0000 7fd07835e640 1 -- 172.21.15.28:0/2184448536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0700b8050 msgr2=0x7fd070160780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.265+0000 7fd07835e640 1 --2- 172.21.15.28:0/2184448536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0700b8050 0x7fd070160780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd0600097a0 tx=0x7fd06002a180 comp rx=0 tx=0).stop 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.265+0000 7fd07835e640 1 -- 172.21.15.28:0/2184448536 shutdown_connections 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.265+0000 7fd07835e640 1 --2- 172.21.15.28:0/2184448536 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0700d4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.265+0000 7fd07835e640 1 --2- 172.21.15.28:0/2184448536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0700b8050 0x7fd070160780 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.265+0000 7fd07835e640 1 -- 172.21.15.28:0/2184448536 >> 172.21.15.28:0/2184448536 conn(0x7fd0700d2980 msgr2=0x7fd0700d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.265+0000 7fd07835e640 1 -- 172.21.15.28:0/2184448536 shutdown_connections 2024-04-25T19:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.265+0000 7fd07835e640 1 -- 172.21.15.28:0/2184448536 wait complete. 2024-04-25T19:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.267+0000 7fd07835e640 1 Processor -- start 2024-04-25T19:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.267+0000 7fd07835e640 1 -- start start 2024-04-25T19:11:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd07835e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd070160120 0x7fd0700cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd07835e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0701461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd07835e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07015aba0 0x7fd070146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd07835e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd070164a10 con 0x7fd070160120 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd07835e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd070164210 con 0x7fd07015aba0 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd07835e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd070144e30 con 0x7fd070160cc0 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd0760fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd070160120 0x7fd0700cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd0760fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd070160120 0x7fd0700cf6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47050/0 (socket says 172.21.15.28:47050) 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd0760fa640 1 -- 172.21.15.28:0/1135307627 learned_addr learned my addr 172.21.15.28:0/1135307627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd0768fb640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07015aba0 0x7fd070146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.268+0000 7fd0758f9640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0701461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd0760fa640 1 -- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 msgr2=0x7fd0701461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd0760fa640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0701461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd0760fa640 1 -- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07015aba0 msgr2=0x7fd070146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd0760fa640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07015aba0 0x7fd070146720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd0760fa640 1 -- 172.21.15.28:0/1135307627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0700cfc30 con 0x7fd070160120 2024-04-25T19:11:50.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd0758f9640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0701461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd0760fa640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd070160120 0x7fd0700cf6f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c440 tx=0x7fd06c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd066ffd640 1 -- 172.21.15.28:0/1135307627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c018070 con 0x7fd070160120 2024-04-25T19:11:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd066ffd640 1 -- 172.21.15.28:0/1135307627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd06c0074e0 con 0x7fd070160120 2024-04-25T19:11:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd070146f80 con 0x7fd070160120 2024-04-25T19:11:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd066ffd640 1 -- 172.21.15.28:0/1135307627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd06c004a20 con 0x7fd070160120 2024-04-25T19:11:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.269+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd07015a0c0 con 0x7fd070160120 2024-04-25T19:11:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.270+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd070050740 con 0x7fd070160120 2024-04-25T19:11:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.271+0000 7fd066ffd640 1 -- 172.21.15.28:0/1135307627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd06c004bc0 con 0x7fd070160120 2024-04-25T19:11:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.271+0000 7fd066ffd640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0540744b0 0x7fd0540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.271+0000 7fd066ffd640 1 -- 172.21.15.28:0/1135307627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd06c094950 con 0x7fd070160120 2024-04-25T19:11:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.271+0000 7fd0758f9640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0540744b0 0x7fd0540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.272+0000 7fd0758f9640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0540744b0 0x7fd0540769c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd060034000 tx=0x7fd06002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:50.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.276+0000 7fd066ffd640 1 -- 172.21.15.28:0/1135307627 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd070160530 con 0x7fd070160120 2024-04-25T19:11:50.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.462+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0700d3f70 con 0x7fd070160120 2024-04-25T19:11:50.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.463+0000 7fd066ffd640 1 -- 172.21.15.28:0/1135307627 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7fd06c05ed20 con 0x7fd070160120 2024-04-25T19:11:50.465 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:50.466 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0540744b0 msgr2=0x7fd0540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0540744b0 0x7fd0540769c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd060034000 tx=0x7fd06002e000 comp rx=0 tx=0).stop 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd070160120 msgr2=0x7fd0700cf6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd070160120 0x7fd0700cf6f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00c440 tx=0x7fd06c00c900 comp rx=0 tx=0).stop 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 shutdown_connections 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd0540744b0 0x7fd0540769c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd07015aba0 0x7fd070146720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd070160cc0 0x7fd0701461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 --2- 172.21.15.28:0/1135307627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd070160120 0x7fd0700cf6f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 >> 172.21.15.28:0/1135307627 conn(0x7fd0700d2980 msgr2=0x7fd0700b4e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 shutdown_connections 2024-04-25T19:11:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.466+0000 7fd07835e640 1 -- 172.21.15.28:0/1135307627 wait complete. 2024-04-25T19:11:50.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T19:11:50.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.592+0000 7f81e1d1f640 1 Processor -- start 2024-04-25T19:11:50.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.592+0000 7f81e1d1f640 1 -- start start 2024-04-25T19:11:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81e1d1f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f81dc0d0ee0 0x7f81dc0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81e1d1f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81dc06f5b0 con 0x7f81dc15aaa0 2024-04-25T19:11:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81e1d1f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81dc0d14a0 con 0x7f81dc0b5020 2024-04-25T19:11:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81e1d1f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f81dc0af290 con 0x7f81dc0d0ee0 2024-04-25T19:11:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81daffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f81dc0d0ee0 0x7f81dc0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81daffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f81dc0d0ee0 0x7f81dc0affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47062/0 (socket says 172.21.15.28:47062) 2024-04-25T19:11:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81db7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f81dc15aaa0 0x7f81dc0b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52368/0 (socket says 172.21.15.28:52368) 2024-04-25T19:11:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81daffd640 1 -- 172.21.15.28:0/526983380 learned_addr learned my addr 172.21.15.28:0/526983380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81daffd640 1 -- 172.21.15.28:0/526983380 >> v1:172.21.15.53:6789/0 conn(0x7f81dc0b5020 legacy=0x7f81dc160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81daffd640 1 -- 172.21.15.28:0/526983380 >> v1:172.21.15.28:6789/0 conn(0x7f81dc15aaa0 legacy=0x7f81dc0b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.594+0000 7f81daffd640 1 -- 172.21.15.28:0/526983380 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81dc0af4e0 con 0x7f81dc0d0ee0 2024-04-25T19:11:50.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.593+0000 7f81da7fc640 1 -- 172.21.15.28:0/526983380 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069975631 0 0) 0x7f81dc06f5b0 con 0x7f81dc15aaa0 2024-04-25T19:11:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.594+0000 7f81daffd640 1 --2- 172.21.15.28:0/526983380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc0d0ee0 0x7f81dc0affc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0af410 tx=0x7f81d002a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493ced77d9221cc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.594+0000 7f81da7fc640 1 -- 172.21.15.28:0/526983380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0037070 con 0x7f81dc0d0ee0 2024-04-25T19:11:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.594+0000 7f81da7fc640 1 -- 172.21.15.28:0/526983380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d0032640 con 0x7f81dc0d0ee0 2024-04-25T19:11:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.594+0000 7f81da7fc640 1 -- 172.21.15.28:0/526983380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0032a10 con 0x7f81dc0d0ee0 2024-04-25T19:11:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.595+0000 7f81e1d1f640 1 -- 172.21.15.28:0/526983380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc0d0ee0 msgr2=0x7f81dc0affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.595+0000 7f81e1d1f640 1 --2- 172.21.15.28:0/526983380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc0d0ee0 0x7f81dc0affc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0af410 tx=0x7f81d002a2e0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.595+0000 7f81e1d1f640 1 -- 172.21.15.28:0/526983380 shutdown_connections 2024-04-25T19:11:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.595+0000 7f81e1d1f640 1 --2- 172.21.15.28:0/526983380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc0d0ee0 0x7f81dc0affc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.595+0000 7f81e1d1f640 1 -- 172.21.15.28:0/526983380 >> 172.21.15.28:0/526983380 conn(0x7f81dc0742c0 msgr2=0x7f81dc06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.595+0000 7f81e1d1f640 1 -- 172.21.15.28:0/526983380 shutdown_connections 2024-04-25T19:11:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.595+0000 7f81e1d1f640 1 -- 172.21.15.28:0/526983380 wait complete. 2024-04-25T19:11:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.597+0000 7f81e1d1f640 1 Processor -- start 2024-04-25T19:11:50.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.597+0000 7f81e1d1f640 1 -- start start 2024-04-25T19:11:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.598+0000 7f81e1d1f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81dc0b5020 0x7f81dc1f4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.598+0000 7f81e1d1f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81dc0d0ee0 0x7f81dc0b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.598+0000 7f81e1d1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc15aaa0 0x7f81dc0b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.598+0000 7f81e1d1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81dc0b5430 con 0x7f81dc15aaa0 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.598+0000 7f81e1d1f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f81dc0afaa0 con 0x7f81dc0b5020 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.598+0000 7f81e1d1f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f81dc0b55b0 con 0x7f81dc0d0ee0 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.598+0000 7f81dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc15aaa0 0x7f81dc0b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81dbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc15aaa0 0x7f81dc0b2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47068/0 (socket says 172.21.15.28:47068) 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81dbfff640 1 -- 172.21.15.28:0/1843187406 learned_addr learned my addr 172.21.15.28:0/1843187406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81db7fe640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81dc0b5020 0x7f81dc1f4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81daffd640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81dc0d0ee0 0x7f81dc0b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81dbfff640 1 -- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81dc0d0ee0 msgr2=0x7f81dc0b2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81dbfff640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81dc0d0ee0 0x7f81dc0b2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81dbfff640 1 -- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81dc0b5020 msgr2=0x7f81dc1f4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81dbfff640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81dc0b5020 0x7f81dc1f4a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81dbfff640 1 -- 172.21.15.28:0/1843187406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81dc1f74f0 con 0x7f81dc15aaa0 2024-04-25T19:11:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81daffd640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81dc0d0ee0 0x7f81dc0b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81dbfff640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc15aaa0 0x7f81dc0b2d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f81d400c6e0 tx=0x7f81d400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81c3fff640 1 -- 172.21.15.28:0/1843187406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d4003b70 con 0x7f81dc15aaa0 2024-04-25T19:11:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81c3fff640 1 -- 172.21.15.28:0/1843187406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d4003d30 con 0x7f81dc15aaa0 2024-04-25T19:11:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81dc1f53d0 con 0x7f81dc15aaa0 2024-04-25T19:11:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81c3fff640 1 -- 172.21.15.28:0/1843187406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d4005310 con 0x7f81dc15aaa0 2024-04-25T19:11:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.599+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81dc1f5990 con 0x7f81dc15aaa0 2024-04-25T19:11:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.600+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81dc1f50d0 con 0x7f81dc15aaa0 2024-04-25T19:11:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.601+0000 7f81c3fff640 1 -- 172.21.15.28:0/1843187406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f81d40054b0 con 0x7f81dc15aaa0 2024-04-25T19:11:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.601+0000 7f81c3fff640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c80744e0 0x7f81c80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.602+0000 7f81c3fff640 1 -- 172.21.15.28:0/1843187406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f81d4095370 con 0x7f81dc15aaa0 2024-04-25T19:11:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.602+0000 7f81db7fe640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c80744e0 0x7f81c80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.602+0000 7f81db7fe640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c80744e0 0x7f81c80769f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f81cc000f60 tx=0x7f81cc007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.606+0000 7f81c3fff640 1 -- 172.21.15.28:0/1843187406 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81d405f7c0 con 0x7f81dc15aaa0 2024-04-25T19:11:50.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:50.832+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f81dc050740 con 0x7f81dc15aaa0 2024-04-25T19:11:51.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.000+0000 7f81c3fff640 1 -- 172.21.15.28:0/1843187406 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f81d405f0c0 con 0x7f81dc15aaa0 2024-04-25T19:11:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c80744e0 msgr2=0x7f81c80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c80744e0 0x7f81c80769f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f81cc000f60 tx=0x7f81cc007e40 comp rx=0 tx=0).stop 2024-04-25T19:11:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc15aaa0 msgr2=0x7f81dc0b2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc15aaa0 0x7f81dc0b2d90 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f81d400c6e0 tx=0x7f81d400cba0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 shutdown_connections 2024-04-25T19:11:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c80744e0 0x7f81c80769f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81dc15aaa0 0x7f81dc0b2d90 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81dc0d0ee0 0x7f81dc0b2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 --2- 172.21.15.28:0/1843187406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81dc0b5020 0x7f81dc1f4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 >> 172.21.15.28:0/1843187406 conn(0x7f81dc0742c0 msgr2=0x7f81dc0cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 shutdown_connections 2024-04-25T19:11:51.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.002+0000 7f81e1d1f640 1 -- 172.21.15.28:0/1843187406 wait complete. 2024-04-25T19:11:51.016 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-25T19:11:51.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T19:11:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.129+0000 7f2b81334640 1 Processor -- start 2024-04-25T19:11:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.129+0000 7f2b81334640 1 -- start start 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.130+0000 7f2b81334640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 0x7f2b7c0d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.130+0000 7f2b81334640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b7c0b79f0 0x7f2b7c0b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b81334640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7c160720 con 0x7f2b7c08cce0 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b81334640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f2b7c0b8340 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b81334640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2b7c0d5640 con 0x7f2b7c160310 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b7a7fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b7c0b79f0 0x7f2b7c0b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b7a7fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b7c0b79f0 0x7f2b7c0b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37084/0 (socket says 172.21.15.28:37084) 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b7a7fc640 1 -- 172.21.15.28:0/2676810434 learned_addr learned my addr 172.21.15.28:0/2676810434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:51.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b7affd640 1 --2- 172.21.15.28:0/2676810434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 0x7f2b7c0d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b7a7fc640 1 -- 172.21.15.28:0/2676810434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 msgr2=0x7f2b7c0d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b7a7fc640 1 --2- 172.21.15.28:0/2676810434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 0x7f2b7c0d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b7a7fc640 1 -- 172.21.15.28:0/2676810434 >> v1:172.21.15.53:6789/0 conn(0x7f2b7c08cce0 legacy=0x7f2b7c08d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:51.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.131+0000 7f2b7a7fc640 1 -- 172.21.15.28:0/2676810434 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7c0d0f40 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.132+0000 7f2b7a7fc640 1 --2- 172.21.15.28:0/2676810434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c0b7e00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009870 tx=0x7f2b6402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba49c8539affa49 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.132+0000 7f2b79ffb640 1 -- 172.21.15.28:0/2676810434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64037070 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.132+0000 7f2b79ffb640 1 -- 172.21.15.28:0/2676810434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b64032440 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.132+0000 7f2b79ffb640 1 -- 172.21.15.28:0/2676810434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64032810 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.133+0000 7f2b81334640 1 -- 172.21.15.28:0/2676810434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 msgr2=0x7f2b7c0b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.133+0000 7f2b81334640 1 --2- 172.21.15.28:0/2676810434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c0b7e00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2b64009870 tx=0x7f2b6402a270 comp rx=0 tx=0).stop 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.133+0000 7f2b81334640 1 -- 172.21.15.28:0/2676810434 shutdown_connections 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.133+0000 7f2b81334640 1 --2- 172.21.15.28:0/2676810434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c0b7e00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.133+0000 7f2b81334640 1 --2- 172.21.15.28:0/2676810434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 0x7f2b7c0d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.133+0000 7f2b81334640 1 -- 172.21.15.28:0/2676810434 >> 172.21.15.28:0/2676810434 conn(0x7f2b7c0d2a50 msgr2=0x7f2b7c0d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.133+0000 7f2b81334640 1 -- 172.21.15.28:0/2676810434 shutdown_connections 2024-04-25T19:11:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.133+0000 7f2b81334640 1 -- 172.21.15.28:0/2676810434 wait complete. 2024-04-25T19:11:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.135+0000 7f2b81334640 1 Processor -- start 2024-04-25T19:11:51.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.135+0000 7f2b81334640 1 -- start start 2024-04-25T19:11:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b81334640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b7c08cce0 0x7f2b7c0ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b81334640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c14ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:51.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b81334640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 0x7f2b7c14b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b81334640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c0b9400 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b81334640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c0d1500 con 0x7f2b7c08cce0 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b81334640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2b7c0d0bf0 con 0x7f2b7c160310 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c14ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c14ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37090/0 (socket says 172.21.15.28:37090) 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7a7fc640 1 -- 172.21.15.28:0/858881834 learned_addr learned my addr 172.21.15.28:0/858881834 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7affd640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b7c08cce0 0x7f2b7c0ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7a7fc640 1 -- 172.21.15.28:0/858881834 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 msgr2=0x7f2b7c14b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7a7fc640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 0x7f2b7c14b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7a7fc640 1 -- 172.21.15.28:0/858881834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b7c08cce0 msgr2=0x7f2b7c0ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7a7fc640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b7c08cce0 0x7f2b7c0ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7a7fc640 1 -- 172.21.15.28:0/858881834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7c0cb0d0 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.136+0000 7f2b7affd640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b7c08cce0 0x7f2b7c0ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.137+0000 7f2b7a7fc640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c14ab90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.137+0000 7f2b737fe640 1 -- 172.21.15.28:0/858881834 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64041020 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.137+0000 7f2b737fe640 1 -- 172.21.15.28:0/858881834 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b640306c0 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.137+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7c0cb3b0 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.137+0000 7f2b737fe640 1 -- 172.21.15.28:0/858881834 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b64037040 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.137+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b7c0cb970 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.138+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b7c15ca00 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.138+0000 7f2b737fe640 1 -- 172.21.15.28:0/858881834 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2b64044020 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.139+0000 7f2b737fe640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2b5c0744e0 0x7f2b5c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.139+0000 7f2b737fe640 1 -- 172.21.15.28:0/858881834 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2b640b5c80 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.139+0000 7f2b7affd640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2b5c0744e0 0x7f2b5c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:51.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.139+0000 7f2b7affd640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2b5c0744e0 0x7f2b5c0769f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b74006fd0 tx=0x7f2b74008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:51.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.144+0000 7f2b737fe640 1 -- 172.21.15.28:0/858881834 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b7c15ca00 con 0x7f2b7c0b79f0 2024-04-25T19:11:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:51.366+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f2b7c0b7e00 con 0x7f2b7c0b79f0 2024-04-25T19:11:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.001+0000 7f2b737fe640 1 -- 172.21.15.28:0/858881834 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f2b6404b030 con 0x7f2b7c0b79f0 2024-04-25T19:11:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2b5c0744e0 msgr2=0x7f2b5c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2b5c0744e0 0x7f2b5c0769f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2b74006fd0 tx=0x7f2b74008040 comp rx=0 tx=0).stop 2024-04-25T19:11:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 msgr2=0x7f2b7c14ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c14ab90 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b640046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 shutdown_connections 2024-04-25T19:11:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2b5c0744e0 0x7f2b5c0769f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2b7c160310 0x7f2b7c14b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b7c0b79f0 0x7f2b7c14ab90 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 --2- 172.21.15.28:0/858881834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b7c08cce0 0x7f2b7c0ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 >> 172.21.15.28:0/858881834 conn(0x7f2b7c0d2a50 msgr2=0x7f2b7c15f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 shutdown_connections 2024-04-25T19:11:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.004+0000 7f2b81334640 1 -- 172.21.15.28:0/858881834 wait complete. 2024-04-25T19:11:52.021 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T19:11:52.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:11:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.139+0000 7f09fdb4e640 1 Processor -- start 2024-04-25T19:11:52.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.139+0000 7f09fdb4e640 1 -- start start 2024-04-25T19:11:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.140+0000 7f09fdb4e640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f09f80b7d70 0x7f09f815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.140+0000 7f09fdb4e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f81606f0 0x7f09f8160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.140+0000 7f09fdb4e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f81614a0 con 0x7f09f8164990 2024-04-25T19:11:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.140+0000 7f09fdb4e640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f09f8163eb0 con 0x7f09f80b7d70 2024-04-25T19:11:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.140+0000 7f09fdb4e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f09f8164190 con 0x7f09f81606f0 2024-04-25T19:11:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.140+0000 7f09f77fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f09f8164990 0x7f09f80b7660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58848/0 (socket says 172.21.15.28:58848) 2024-04-25T19:11:52.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.140+0000 7f09f77fe640 1 -- 172.21.15.28:0/3693156729 learned_addr learned my addr 172.21.15.28:0/3693156729 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f6ffd640 1 --2- 172.21.15.28:0/3693156729 >> v2:172.21.15.53:3300/0 conn(0x7f09f80b7d70 0x7f09f815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f7fff640 1 --2- 172.21.15.28:0/3693156729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f81606f0 0x7f09f8160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f67fc640 1 -- 172.21.15.28:0/3693156729 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2329167105 0 0) 0x7f09f81614a0 con 0x7f09f8164990 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f67fc640 1 -- 172.21.15.28:0/3693156729 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09e4003520 con 0x7f09f8164990 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f7fff640 1 -- 172.21.15.28:0/3693156729 >> v2:172.21.15.53:3300/0 conn(0x7f09f80b7d70 msgr2=0x7f09f815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f7fff640 1 --2- 172.21.15.28:0/3693156729 >> v2:172.21.15.53:3300/0 conn(0x7f09f80b7d70 0x7f09f815dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f7fff640 1 -- 172.21.15.28:0/3693156729 >> v1:172.21.15.28:6789/0 conn(0x7f09f8164990 legacy=0x7f09f80b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f67fc640 1 -- 172.21.15.28:0/3693156729 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1372054358 0 0) 0x7f09e4003520 con 0x7f09f8164990 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.141+0000 7f09f7fff640 1 -- 172.21.15.28:0/3693156729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f8164470 con 0x7f09f81606f0 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.142+0000 7f09f7fff640 1 --2- 172.21.15.28:0/3693156729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f81606f0 0x7f09f8160b00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009d90 tx=0x7f09f002a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e2657ad15f4ea4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.142+0000 7f09f67fc640 1 -- 172.21.15.28:0/3693156729 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0001690 con 0x7f09f81606f0 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.142+0000 7f09f67fc640 1 -- 172.21.15.28:0/3693156729 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09f0007540 con 0x7f09f81606f0 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.142+0000 7f09f67fc640 1 -- 172.21.15.28:0/3693156729 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f00078f0 con 0x7f09f81606f0 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.143+0000 7f09fdb4e640 1 -- 172.21.15.28:0/3693156729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f81606f0 msgr2=0x7f09f8160b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.143+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/3693156729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f81606f0 0x7f09f8160b00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009d90 tx=0x7f09f002a5f0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.143+0000 7f09fdb4e640 1 -- 172.21.15.28:0/3693156729 shutdown_connections 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.143+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/3693156729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f81606f0 0x7f09f8160b00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.143+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/3693156729 >> v2:172.21.15.53:3300/0 conn(0x7f09f80b7d70 0x7f09f815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.143+0000 7f09fdb4e640 1 -- 172.21.15.28:0/3693156729 >> 172.21.15.28:0/3693156729 conn(0x7f09f80d2840 msgr2=0x7f09f80d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.143+0000 7f09fdb4e640 1 -- 172.21.15.28:0/3693156729 shutdown_connections 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.143+0000 7f09fdb4e640 1 -- 172.21.15.28:0/3693156729 wait complete. 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.144+0000 7f09fdb4e640 1 Processor -- start 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.145+0000 7f09fdb4e640 1 -- start start 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09fdb4e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f80b7d70 0x7f09f8152e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09fdb4e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81606f0 0x7f09f80ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09fdb4e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f8164990 0x7f09f80cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09fdb4e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09f80b4750 con 0x7f09f81606f0 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09fdb4e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f09f80b45d0 con 0x7f09f8164990 2024-04-25T19:11:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09fdb4e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f09f80b48d0 con 0x7f09f80b7d70 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81606f0 0x7f09f80ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09f6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81606f0 0x7f09f80ccc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37094/0 (socket says 172.21.15.28:37094) 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09f6ffd640 1 -- 172.21.15.28:0/1859686590 learned_addr learned my addr 172.21.15.28:0/1859686590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09f7fff640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f8164990 0x7f09f80cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09f77fe640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f80b7d70 0x7f09f8152e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09f6ffd640 1 -- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f80b7d70 msgr2=0x7f09f8152e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09f6ffd640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f80b7d70 0x7f09f8152e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.146+0000 7f09f6ffd640 1 -- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f8164990 msgr2=0x7f09f80cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09f6ffd640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f8164990 0x7f09f80cdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09f6ffd640 1 -- 172.21.15.28:0/1859686590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f80ba350 con 0x7f09f81606f0 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09f7fff640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f8164990 0x7f09f80cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09f6ffd640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81606f0 0x7f09f80ccc90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09e800e8a0 tx=0x7f09e800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09effff640 1 -- 172.21.15.28:0/1859686590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8017070 con 0x7f09f81606f0 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09effff640 1 -- 172.21.15.28:0/1859686590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e800c940 con 0x7f09f81606f0 2024-04-25T19:11:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09effff640 1 -- 172.21.15.28:0/1859686590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e800cd30 con 0x7f09f81606f0 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f80ba7c0 con 0x7f09f81606f0 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.147+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f80caff0 con 0x7f09f81606f0 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.148+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f80ba520 con 0x7f09f81606f0 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.149+0000 7f09effff640 1 -- 172.21.15.28:0/1859686590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f09e801a070 con 0x7f09f81606f0 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.149+0000 7f09effff640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09d80744b0 0x7f09d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.149+0000 7f09effff640 1 -- 172.21.15.28:0/1859686590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f09e8095380 con 0x7f09f81606f0 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.149+0000 7f09f77fe640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09d80744b0 0x7f09d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.150+0000 7f09f77fe640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09d80744b0 0x7f09d80769c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09e0001200 tx=0x7f09e00016f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.154+0000 7f09effff640 1 -- 172.21.15.28:0/1859686590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09f80ba520 con 0x7f09f81606f0 2024-04-25T19:11:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.379+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09f8050740 con 0x7f09f81606f0 2024-04-25T19:11:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.379+0000 7f09effff640 1 -- 172.21.15.28:0/1859686590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f09e805f190 con 0x7f09f81606f0 2024-04-25T19:11:52.381 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:52.381 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:51.367677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:11:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T19:11:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09d80744b0 msgr2=0x7f09d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09d80744b0 0x7f09d80769c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09e0001200 tx=0x7f09e00016f0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81606f0 msgr2=0x7f09f80ccc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81606f0 0x7f09f80ccc90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f09e800e8a0 tx=0x7f09e800ed60 comp rx=0 tx=0).stop 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 shutdown_connections 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09d80744b0 0x7f09d80769c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09f8164990 0x7f09f80cdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81606f0 0x7f09f80ccc90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 --2- 172.21.15.28:0/1859686590 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09f80b7d70 0x7f09f8152e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 >> 172.21.15.28:0/1859686590 conn(0x7f09f80d2840 msgr2=0x7f09f8161c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 shutdown_connections 2024-04-25T19:11:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.381+0000 7f09fdb4e640 1 -- 172.21.15.28:0/1859686590 wait complete. 2024-04-25T19:11:52.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.504+0000 7f057bfc7640 1 Processor -- start 2024-04-25T19:11:52.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.504+0000 7f057bfc7640 1 -- start start 2024-04-25T19:11:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057bfc7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 0x7f0574161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057bfc7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f057415f7d0 0x7f057415fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057bfc7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0574163e80 con 0x7f057406ded0 2024-04-25T19:11:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057bfc7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f0574163c30 con 0x7f057415f7d0 2024-04-25T19:11:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057bfc7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0574164a90 con 0x7f05740b5130 2024-04-25T19:11:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057a564640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f057415f7d0 0x7f057415fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f0579562640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f057406ded0 0x7f057406e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58862/0 (socket says 172.21.15.28:58862) 2024-04-25T19:11:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f0579562640 1 -- 172.21.15.28:0/3604590222 learned_addr learned my addr 172.21.15.28:0/3604590222 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f0579d63640 1 --2- 172.21.15.28:0/3604590222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 0x7f0574161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057a564640 1 -- 172.21.15.28:0/3604590222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 msgr2=0x7f0574161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057a564640 1 --2- 172.21.15.28:0/3604590222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 0x7f0574161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057a564640 1 -- 172.21.15.28:0/3604590222 >> v1:172.21.15.28:6789/0 conn(0x7f057406ded0 legacy=0x7f057406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f057a564640 1 -- 172.21.15.28:0/3604590222 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0574164160 con 0x7f057415f7d0 2024-04-25T19:11:52.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.506+0000 7f0578d61640 1 -- 172.21.15.28:0/3604590222 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3473547767 0 0) 0x7f0574163e80 con 0x7f057406ded0 2024-04-25T19:11:52.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.507+0000 7f057a564640 1 --2- 172.21.15.28:0/3604590222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057415f7d0 0x7f057415fbe0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f056c0097a0 tx=0x7f056c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b7701c7968cdf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.507+0000 7f0578d61640 1 -- 172.21.15.28:0/3604590222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c037070 con 0x7f057415f7d0 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.507+0000 7f0578d61640 1 -- 172.21.15.28:0/3604590222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f056c032440 con 0x7f057415f7d0 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.507+0000 7f0578d61640 1 -- 172.21.15.28:0/3604590222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056c032810 con 0x7f057415f7d0 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.508+0000 7f057bfc7640 1 -- 172.21.15.28:0/3604590222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057415f7d0 msgr2=0x7f057415fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.508+0000 7f057bfc7640 1 --2- 172.21.15.28:0/3604590222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057415f7d0 0x7f057415fbe0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f056c0097a0 tx=0x7f056c02a180 comp rx=0 tx=0).stop 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.508+0000 7f057bfc7640 1 -- 172.21.15.28:0/3604590222 shutdown_connections 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.508+0000 7f057bfc7640 1 --2- 172.21.15.28:0/3604590222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057415f7d0 0x7f057415fbe0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.508+0000 7f057bfc7640 1 --2- 172.21.15.28:0/3604590222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 0x7f0574161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.508+0000 7f057bfc7640 1 -- 172.21.15.28:0/3604590222 >> 172.21.15.28:0/3604590222 conn(0x7f05740b7ad0 msgr2=0x7f05740b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.508+0000 7f057bfc7640 1 -- 172.21.15.28:0/3604590222 shutdown_connections 2024-04-25T19:11:52.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.508+0000 7f057bfc7640 1 -- 172.21.15.28:0/3604590222 wait complete. 2024-04-25T19:11:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.510+0000 7f057bfc7640 1 Processor -- start 2024-04-25T19:11:52.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.510+0000 7f057bfc7640 1 -- start start 2024-04-25T19:11:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f057bfc7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057406ded0 0x7f057415a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f057bfc7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 0x7f05741622b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f057bfc7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f7d0 0x7f05741627f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f057bfc7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0574160120 con 0x7f057406ded0 2024-04-25T19:11:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f057bfc7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0574164720 con 0x7f057415f7d0 2024-04-25T19:11:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f057bfc7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f05741602a0 con 0x7f05740b5130 2024-04-25T19:11:52.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579d63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057406ded0 0x7f057415a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579d63640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057406ded0 0x7f057415a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37114/0 (socket says 172.21.15.28:37114) 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579d63640 1 -- 172.21.15.28:0/1736370206 learned_addr learned my addr 172.21.15.28:0/1736370206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579562640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 0x7f05741622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f057a564640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f7d0 0x7f05741627f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579d63640 1 -- 172.21.15.28:0/1736370206 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 msgr2=0x7f05741622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579d63640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 0x7f05741622b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579d63640 1 -- 172.21.15.28:0/1736370206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f7d0 msgr2=0x7f05741627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579d63640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f7d0 0x7f05741627f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.511+0000 7f0579d63640 1 -- 172.21.15.28:0/1736370206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057414a160 con 0x7f057406ded0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.512+0000 7f0579d63640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057406ded0 0x7f057415a700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0570009cd0 tx=0x7f057000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.512+0000 7f056a7fc640 1 -- 172.21.15.28:0/1736370206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570018020 con 0x7f057406ded0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.512+0000 7f056a7fc640 1 -- 172.21.15.28:0/1736370206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0570014480 con 0x7f057406ded0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.512+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0574147550 con 0x7f057406ded0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.512+0000 7f056a7fc640 1 -- 172.21.15.28:0/1736370206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0570014850 con 0x7f057406ded0 2024-04-25T19:11:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.512+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05741490b0 con 0x7f057406ded0 2024-04-25T19:11:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.513+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0574050740 con 0x7f057406ded0 2024-04-25T19:11:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.514+0000 7f056a7fc640 1 -- 172.21.15.28:0/1736370206 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0570005d10 con 0x7f057406ded0 2024-04-25T19:11:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.514+0000 7f056a7fc640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05580744e0 0x7f05580769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.514+0000 7f056a7fc640 1 -- 172.21.15.28:0/1736370206 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f05700962d0 con 0x7f057406ded0 2024-04-25T19:11:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.514+0000 7f0579562640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05580744e0 0x7f05580769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.515+0000 7f0579562640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05580744e0 0x7f05580769f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0574157790 tx=0x7f0564009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.519+0000 7f056a7fc640 1 -- 172.21.15.28:0/1736370206 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0570022030 con 0x7f057406ded0 2024-04-25T19:11:52.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.703+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0574157d40 con 0x7f057406ded0 2024-04-25T19:11:52.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.705+0000 7f056a7fc640 1 -- 172.21.15.28:0/1736370206 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f0574157d40 con 0x7f057406ded0 2024-04-25T19:11:52.707 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:52.707 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05580744e0 msgr2=0x7f05580769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05580744e0 0x7f05580769f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0574157790 tx=0x7f0564009040 comp rx=0 tx=0).stop 2024-04-25T19:11:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057406ded0 msgr2=0x7f057415a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057406ded0 0x7f057415a700 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0570009cd0 tx=0x7f057000c490 comp rx=0 tx=0).stop 2024-04-25T19:11:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 shutdown_connections 2024-04-25T19:11:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05580744e0 0x7f05580769f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f7d0 0x7f05741627f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05740b5130 0x7f05741622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 --2- 172.21.15.28:0/1736370206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057406ded0 0x7f057415a700 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 >> 172.21.15.28:0/1736370206 conn(0x7f05740b7ad0 msgr2=0x7f05740d4980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 shutdown_connections 2024-04-25T19:11:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.707+0000 7f057bfc7640 1 -- 172.21.15.28:0/1736370206 wait complete. 2024-04-25T19:11:52.720 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T19:11:52.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:11:52.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.832+0000 7efc4704c640 1 Processor -- start 2024-04-25T19:11:52.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.832+0000 7efc4704c640 1 -- start start 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.833+0000 7efc4704c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efc400d4910 0x7efc40161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.833+0000 7efc4704c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc40163e80 con 0x7efc400b5130 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.833+0000 7efc4704c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc40164160 con 0x7efc4006e5e0 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.833+0000 7efc4704c640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7efc40163c30 con 0x7efc400d4910 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.833+0000 7efc44de8640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efc400b5130 0x7efc4006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58872/0 (socket says 172.21.15.28:58872) 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.833+0000 7efc44de8640 1 -- 172.21.15.28:0/3887777201 learned_addr learned my addr 172.21.15.28:0/3887777201 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.833+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1609893961 0 0) 0x7efc40163e80 con 0x7efc400b5130 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc2c003520 con 0x7efc400b5130 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc455e9640 1 --2- 172.21.15.28:0/3887777201 >> v2:172.21.15.53:3300/0 conn(0x7efc400d4910 0x7efc40161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1745554428 0 0) 0x7efc2c003520 con 0x7efc400b5130 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 >> v2:172.21.15.53:3300/0 conn(0x7efc400d4910 msgr2=0x7efc40161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:52.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 --2- 172.21.15.28:0/3887777201 >> v2:172.21.15.53:3300/0 conn(0x7efc400d4910 0x7efc40161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 >> v1:172.21.15.53:6789/0 conn(0x7efc4006e5e0 legacy=0x7efc4015f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc40164440 con 0x7efc400b5130 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7efc380019f0 con 0x7efc400b5130 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 >> v1:172.21.15.28:6789/0 conn(0x7efc400b5130 legacy=0x7efc4006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc2c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc2c004800 0x7efc2c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc2c0051c0 0x7efc2c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc40163c30 con 0x7efc2c0039b0 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efc2c00c160 con 0x7efc2c0051c0 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efc2c00c360 con 0x7efc2c004800 2024-04-25T19:11:52.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc44de8640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc2c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc3ffff640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc2c004800 0x7efc2c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc455e9640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc2c0051c0 0x7efc2c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc44de8640 1 -- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc2c004800 msgr2=0x7efc2c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc44de8640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc2c004800 0x7efc2c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc44de8640 1 -- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc2c0051c0 msgr2=0x7efc2c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc44de8640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc2c0051c0 0x7efc2c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc44de8640 1 -- 172.21.15.28:0/3887777201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2c003520 con 0x7efc2c0039b0 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc455e9640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc2c0051c0 0x7efc2c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.834+0000 7efc455e9640 1 -- 172.21.15.28:0/3887777201 reap_dead start 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.835+0000 7efc44de8640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc2c003e20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc3800a830 tx=0x7efc2c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7108ba6623e9cfac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.835+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3800a480 con 0x7efc2c0039b0 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.835+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc3802f070 con 0x7efc2c0039b0 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.835+0000 7efc3f7fe640 1 -- 172.21.15.28:0/3887777201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3802bc20 con 0x7efc2c0039b0 2024-04-25T19:11:52.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.836+0000 7efc4704c640 1 -- 172.21.15.28:0/3887777201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 msgr2=0x7efc2c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.836+0000 7efc4704c640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc2c003e20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc3800a830 tx=0x7efc2c005830 comp rx=0 tx=0).stop 2024-04-25T19:11:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.836+0000 7efc4704c640 1 -- 172.21.15.28:0/3887777201 shutdown_connections 2024-04-25T19:11:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.836+0000 7efc4704c640 1 --2- 172.21.15.28:0/3887777201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc2c003e20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.836+0000 7efc4704c640 1 -- 172.21.15.28:0/3887777201 >> 172.21.15.28:0/3887777201 conn(0x7efc400b7ad0 msgr2=0x7efc400b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.836+0000 7efc4704c640 1 -- 172.21.15.28:0/3887777201 shutdown_connections 2024-04-25T19:11:52.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.836+0000 7efc4704c640 1 -- 172.21.15.28:0/3887777201 wait complete. 2024-04-25T19:11:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.838+0000 7efc4704c640 1 Processor -- start 2024-04-25T19:11:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.838+0000 7efc4704c640 1 -- start start 2024-04-25T19:11:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.839+0000 7efc4704c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc4014af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.839+0000 7efc4704c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc401588b0 0x7efc401f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.839+0000 7efc4704c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc4015bcd0 0x7efc4006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.839+0000 7efc4704c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc40161130 con 0x7efc2c0039b0 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.839+0000 7efc4704c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efc40164a90 con 0x7efc4015bcd0 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.839+0000 7efc4704c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efc401612b0 con 0x7efc401588b0 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc4014af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc4014af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37132/0 (socket says 172.21.15.28:37132) 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 -- 172.21.15.28:0/1206216163 learned_addr learned my addr 172.21.15.28:0/1206216163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc3ffff640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc401588b0 0x7efc401f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc455e9640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc4015bcd0 0x7efc4006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 -- 172.21.15.28:0/1206216163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc401588b0 msgr2=0x7efc401f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc401588b0 0x7efc401f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 -- 172.21.15.28:0/1206216163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc4015bcd0 msgr2=0x7efc4006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc4015bcd0 0x7efc4006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:52.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 -- 172.21.15.28:0/1206216163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc4014b4b0 con 0x7efc2c0039b0 2024-04-25T19:11:52.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc44de8640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc4014af70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efc3800a890 tx=0x7efc40164a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc3d7fa640 1 -- 172.21.15.28:0/1206216163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc38044070 con 0x7efc2c0039b0 2024-04-25T19:11:52.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc3d7fa640 1 -- 172.21.15.28:0/1206216163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc3802bd20 con 0x7efc2c0039b0 2024-04-25T19:11:52.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc401573a0 con 0x7efc2c0039b0 2024-04-25T19:11:52.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc3d7fa640 1 -- 172.21.15.28:0/1206216163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc3800a480 con 0x7efc2c0039b0 2024-04-25T19:11:52.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.840+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc40157af0 con 0x7efc2c0039b0 2024-04-25T19:11:52.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.842+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc40050740 con 0x7efc2c0039b0 2024-04-25T19:11:52.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.842+0000 7efc3d7fa640 1 -- 172.21.15.28:0/1206216163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efc38047070 con 0x7efc2c0039b0 2024-04-25T19:11:52.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.843+0000 7efc3d7fa640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc200744b0 0x7efc200769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:52.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.843+0000 7efc3d7fa640 1 -- 172.21.15.28:0/1206216163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efc380b5260 con 0x7efc2c0039b0 2024-04-25T19:11:52.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.843+0000 7efc3ffff640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc200744b0 0x7efc200769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:52.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.843+0000 7efc3ffff640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc200744b0 0x7efc200769c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efc401f9630 tx=0x7efc28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:52.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:52.847+0000 7efc3d7fa640 1 -- 172.21.15.28:0/1206216163 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc4014b4b0 con 0x7efc2c0039b0 2024-04-25T19:11:53.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.034+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efc400d2fe0 con 0x7efc200744b0 2024-04-25T19:11:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.043+0000 7efc3d7fa640 1 -- 172.21.15.28:0/1206216163 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7efc400d2fe0 con 0x7efc200744b0 2024-04-25T19:11:53.049 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:53.051 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc200744b0 msgr2=0x7efc200769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc200744b0 0x7efc200769c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7efc401f9630 tx=0x7efc28009040 comp rx=0 tx=0).stop 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 msgr2=0x7efc4014af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc4014af70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7efc3800a890 tx=0x7efc40164a00 comp rx=0 tx=0).stop 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 shutdown_connections 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc200744b0 0x7efc200769c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc4015bcd0 0x7efc4006ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc401588b0 0x7efc401f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 --2- 172.21.15.28:0/1206216163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc2c0039b0 0x7efc4014af70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 >> 172.21.15.28:0/1206216163 conn(0x7efc400b7ad0 msgr2=0x7efc40163040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 shutdown_connections 2024-04-25T19:11:53.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.047+0000 7efc4704c640 1 -- 172.21.15.28:0/1206216163 wait complete. 2024-04-25T19:11:53.060 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-25T19:11:52.841746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252656,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722064,"statfs":{"total":773094113280,"available":772835393536,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003131"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878592+0000","last_change":"2024-04-25T19:11:28.938792+0000","last_active":"2024-04-25T19:11:35.878592+0000","last_peered":"2024-04-25T19:11:35.878592+0000","last_clean":"2024-04-25T19:11:35.878592+0000","last_became_active":"2024-04-25T19:11:28.938417+0000","last_became_peered":"2024-04-25T19:11:28.938417+0000","last_unstale":"2024-04-25T19:11:35.878592+0000","last_undegraded":"2024-04-25T19:11:35.878592+0000","last_fullsized":"2024-04-25T19:11:35.878592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:44.780178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171775+0000","last_change":"2024-04-25T19:11:29.937823+0000","last_active":"2024-04-25T19:11:37.171775+0000","last_peered":"2024-04-25T19:11:37.171775+0000","last_clean":"2024-04-25T19:11:37.171775+0000","last_became_active":"2024-04-25T19:11:29.937501+0000","last_became_peered":"2024-04-25T19:11:29.937501+0000","last_unstale":"2024-04-25T19:11:37.171775+0000","last_undegraded":"2024-04-25T19:11:37.171775+0000","last_fullsized":"2024-04-25T19:11:37.171775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170976+0000","last_change":"2024-04-25T19:11:34.113864+0000","last_active":"2024-04-25T19:11:37.170976+0000","last_peered":"2024-04-25T19:11:37.170976+0000","last_clean":"2024-04-25T19:11:37.170976+0000","last_became_active":"2024-04-25T19:11:34.113449+0000","last_became_peered":"2024-04-25T19:11:34.113449+0000","last_unstale":"2024-04-25T19:11:37.170976+0000","last_undegraded":"2024-04-25T19:11:37.170976+0000","last_fullsized":"2024-04-25T19:11:37.170976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.247905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170712+0000","last_change":"2024-04-25T19:11:28.934472+0000","last_active":"2024-04-25T19:11:37.170712+0000","last_peered":"2024-04-25T19:11:37.170712+0000","last_clean":"2024-04-25T19:11:37.170712+0000","last_became_active":"2024-04-25T19:11:28.934016+0000","last_became_peered":"2024-04-25T19:11:28.934016+0000","last_unstale":"2024-04-25T19:11:37.170712+0000","last_undegraded":"2024-04-25T19:11:37.170712+0000","last_fullsized":"2024-04-25T19:11:37.170712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:27:30.904833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878257+0000","last_change":"2024-04-25T19:11:29.944065+0000","last_active":"2024-04-25T19:11:35.878257+0000","last_peered":"2024-04-25T19:11:35.878257+0000","last_clean":"2024-04-25T19:11:35.878257+0000","last_became_active":"2024-04-25T19:11:29.943773+0000","last_became_peered":"2024-04-25T19:11:29.943773+0000","last_unstale":"2024-04-25T19:11:35.878257+0000","last_undegraded":"2024-04-25T19:11:35.878257+0000","last_fullsized":"2024-04-25T19:11:35.878257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:15.552217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989732+0000","last_change":"2024-04-25T19:11:33.975423+0000","last_active":"2024-04-25T19:11:36.989732+0000","last_peered":"2024-04-25T19:11:36.989732+0000","last_clean":"2024-04-25T19:11:36.989732+0000","last_became_active":"2024-04-25T19:11:33.974878+0000","last_became_peered":"2024-04-25T19:11:33.974878+0000","last_unstale":"2024-04-25T19:11:36.989732+0000","last_undegraded":"2024-04-25T19:11:36.989732+0000","last_fullsized":"2024-04-25T19:11:36.989732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:23.927981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170963+0000","last_change":"2024-04-25T19:11:28.937102+0000","last_active":"2024-04-25T19:11:37.170963+0000","last_peered":"2024-04-25T19:11:37.170963+0000","last_clean":"2024-04-25T19:11:37.170963+0000","last_became_active":"2024-04-25T19:11:28.936699+0000","last_became_peered":"2024-04-25T19:11:28.936699+0000","last_unstale":"2024-04-25T19:11:37.170963+0000","last_undegraded":"2024-04-25T19:11:37.170963+0000","last_fullsized":"2024-04-25T19:11:37.170963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:38.236609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962397+0000","last_change":"2024-04-25T19:11:29.947706+0000","last_active":"2024-04-25T19:11:34.962397+0000","last_peered":"2024-04-25T19:11:34.962397+0000","last_clean":"2024-04-25T19:11:34.962397+0000","last_became_active":"2024-04-25T19:11:29.945963+0000","last_became_peered":"2024-04-25T19:11:29.945963+0000","last_unstale":"2024-04-25T19:11:34.962397+0000","last_undegraded":"2024-04-25T19:11:34.962397+0000","last_fullsized":"2024-04-25T19:11:34.962397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:51.175553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990390+0000","last_change":"2024-04-25T19:11:33.975724+0000","last_active":"2024-04-25T19:11:36.990390+0000","last_peered":"2024-04-25T19:11:36.990390+0000","last_clean":"2024-04-25T19:11:36.990390+0000","last_became_active":"2024-04-25T19:11:33.974921+0000","last_became_peered":"2024-04-25T19:11:33.974921+0000","last_unstale":"2024-04-25T19:11:36.990390+0000","last_undegraded":"2024-04-25T19:11:36.990390+0000","last_fullsized":"2024-04-25T19:11:36.990390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:55.208890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170918+0000","last_change":"2024-04-25T19:11:28.940100+0000","last_active":"2024-04-25T19:11:37.170918+0000","last_peered":"2024-04-25T19:11:37.170918+0000","last_clean":"2024-04-25T19:11:37.170918+0000","last_became_active":"2024-04-25T19:11:28.939482+0000","last_became_peered":"2024-04-25T19:11:28.939482+0000","last_unstale":"2024-04-25T19:11:37.170918+0000","last_undegraded":"2024-04-25T19:11:37.170918+0000","last_fullsized":"2024-04-25T19:11:37.170918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962428+0000","last_change":"2024-04-25T19:11:29.945900+0000","last_active":"2024-04-25T19:11:34.962428+0000","last_peered":"2024-04-25T19:11:34.962428+0000","last_clean":"2024-04-25T19:11:34.962428+0000","last_became_active":"2024-04-25T19:11:29.945554+0000","last_became_peered":"2024-04-25T19:11:29.945554+0000","last_unstale":"2024-04-25T19:11:34.962428+0000","last_undegraded":"2024-04-25T19:11:34.962428+0000","last_fullsized":"2024-04-25T19:11:34.962428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990011+0000","last_change":"2024-04-25T19:11:33.975263+0000","last_active":"2024-04-25T19:11:36.990011+0000","last_peered":"2024-04-25T19:11:36.990011+0000","last_clean":"2024-04-25T19:11:36.990011+0000","last_became_active":"2024-04-25T19:11:33.974661+0000","last_became_peered":"2024-04-25T19:11:33.974661+0000","last_unstale":"2024-04-25T19:11:36.990011+0000","last_undegraded":"2024-04-25T19:11:36.990011+0000","last_fullsized":"2024-04-25T19:11:36.990011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:29.019166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961851+0000","last_change":"2024-04-25T19:11:29.939828+0000","last_active":"2024-04-25T19:11:34.961851+0000","last_peered":"2024-04-25T19:11:34.961851+0000","last_clean":"2024-04-25T19:11:34.961851+0000","last_became_active":"2024-04-25T19:11:29.939513+0000","last_became_peered":"2024-04-25T19:11:29.939513+0000","last_unstale":"2024-04-25T19:11:34.961851+0000","last_undegraded":"2024-04-25T19:11:34.961851+0000","last_fullsized":"2024-04-25T19:11:34.961851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171950+0000","last_change":"2024-04-25T19:11:28.937339+0000","last_active":"2024-04-25T19:11:37.171950+0000","last_peered":"2024-04-25T19:11:37.171950+0000","last_clean":"2024-04-25T19:11:37.171950+0000","last_became_active":"2024-04-25T19:11:28.936962+0000","last_became_peered":"2024-04-25T19:11:28.936962+0000","last_unstale":"2024-04-25T19:11:37.171950+0000","last_undegraded":"2024-04-25T19:11:37.171950+0000","last_fullsized":"2024-04-25T19:11:37.171950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:30.900062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990657+0000","last_change":"2024-04-25T19:11:33.975365+0000","last_active":"2024-04-25T19:11:36.990657+0000","last_peered":"2024-04-25T19:11:36.990657+0000","last_clean":"2024-04-25T19:11:36.990657+0000","last_became_active":"2024-04-25T19:11:33.974761+0000","last_became_peered":"2024-04-25T19:11:33.974761+0000","last_unstale":"2024-04-25T19:11:36.990657+0000","last_undegraded":"2024-04-25T19:11:36.990657+0000","last_fullsized":"2024-04-25T19:11:36.990657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:46.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171339+0000","last_change":"2024-04-25T19:11:29.935255+0000","last_active":"2024-04-25T19:11:37.171339+0000","last_peered":"2024-04-25T19:11:37.171339+0000","last_clean":"2024-04-25T19:11:37.171339+0000","last_became_active":"2024-04-25T19:11:29.934911+0000","last_became_peered":"2024-04-25T19:11:29.934911+0000","last_unstale":"2024-04-25T19:11:37.171339+0000","last_undegraded":"2024-04-25T19:11:37.171339+0000","last_fullsized":"2024-04-25T19:11:37.171339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470651+0000","last_change":"2024-04-25T19:11:28.937507+0000","last_active":"2024-04-25T19:11:35.470651+0000","last_peered":"2024-04-25T19:11:35.470651+0000","last_clean":"2024-04-25T19:11:35.470651+0000","last_became_active":"2024-04-25T19:11:28.936655+0000","last_became_peered":"2024-04-25T19:11:28.936655+0000","last_unstale":"2024-04-25T19:11:35.470651+0000","last_undegraded":"2024-04-25T19:11:35.470651+0000","last_fullsized":"2024-04-25T19:11:35.470651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990392+0000","last_change":"2024-04-25T19:11:33.974529+0000","last_active":"2024-04-25T19:11:36.990392+0000","last_peered":"2024-04-25T19:11:36.990392+0000","last_clean":"2024-04-25T19:11:36.990392+0000","last_became_active":"2024-04-25T19:11:33.973907+0000","last_became_peered":"2024-04-25T19:11:33.973907+0000","last_unstale":"2024-04-25T19:11:36.990392+0000","last_undegraded":"2024-04-25T19:11:36.990392+0000","last_fullsized":"2024-04-25T19:11:36.990392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:52.041722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171394+0000","last_change":"2024-04-25T19:11:29.930361+0000","last_active":"2024-04-25T19:11:37.171394+0000","last_peered":"2024-04-25T19:11:37.171394+0000","last_clean":"2024-04-25T19:11:37.171394+0000","last_became_active":"2024-04-25T19:11:29.930060+0000","last_became_peered":"2024-04-25T19:11:29.930060+0000","last_unstale":"2024-04-25T19:11:37.171394+0000","last_undegraded":"2024-04-25T19:11:37.171394+0000","last_fullsized":"2024-04-25T19:11:37.171394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470271+0000","last_change":"2024-04-25T19:11:28.930410+0000","last_active":"2024-04-25T19:11:35.470271+0000","last_peered":"2024-04-25T19:11:35.470271+0000","last_clean":"2024-04-25T19:11:35.470271+0000","last_became_active":"2024-04-25T19:11:28.930012+0000","last_became_peered":"2024-04-25T19:11:28.930012+0000","last_unstale":"2024-04-25T19:11:35.470271+0000","last_undegraded":"2024-04-25T19:11:35.470271+0000","last_fullsized":"2024-04-25T19:11:35.470271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.471102+0000","last_change":"2024-04-25T19:11:34.115931+0000","last_active":"2024-04-25T19:11:35.471102+0000","last_peered":"2024-04-25T19:11:35.471102+0000","last_clean":"2024-04-25T19:11:35.471102+0000","last_became_active":"2024-04-25T19:11:34.115519+0000","last_became_peered":"2024-04-25T19:11:34.115519+0000","last_unstale":"2024-04-25T19:11:35.471102+0000","last_undegraded":"2024-04-25T19:11:35.471102+0000","last_fullsized":"2024-04-25T19:11:35.471102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.795134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990163+0000","last_change":"2024-04-25T19:11:29.930630+0000","last_active":"2024-04-25T19:11:36.990163+0000","last_peered":"2024-04-25T19:11:36.990163+0000","last_clean":"2024-04-25T19:11:36.990163+0000","last_became_active":"2024-04-25T19:11:29.930314+0000","last_became_peered":"2024-04-25T19:11:29.930314+0000","last_unstale":"2024-04-25T19:11:36.990163+0000","last_undegraded":"2024-04-25T19:11:36.990163+0000","last_fullsized":"2024-04-25T19:11:36.990163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:44.955421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170921+0000","last_change":"2024-04-25T19:11:28.938687+0000","last_active":"2024-04-25T19:11:37.170921+0000","last_peered":"2024-04-25T19:11:37.170921+0000","last_clean":"2024-04-25T19:11:37.170921+0000","last_became_active":"2024-04-25T19:11:28.938288+0000","last_became_peered":"2024-04-25T19:11:28.938288+0000","last_unstale":"2024-04-25T19:11:37.170921+0000","last_undegraded":"2024-04-25T19:11:37.170921+0000","last_fullsized":"2024-04-25T19:11:37.170921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990666+0000","last_change":"2024-04-25T19:11:33.972056+0000","last_active":"2024-04-25T19:11:36.990666+0000","last_peered":"2024-04-25T19:11:36.990666+0000","last_clean":"2024-04-25T19:11:36.990666+0000","last_became_active":"2024-04-25T19:11:33.971484+0000","last_became_peered":"2024-04-25T19:11:33.971484+0000","last_unstale":"2024-04-25T19:11:36.990666+0000","last_undegraded":"2024-04-25T19:11:36.990666+0000","last_fullsized":"2024-04-25T19:11:36.990666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:52.951275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878644+0000","last_change":"2024-04-25T19:11:29.935281+0000","last_active":"2024-04-25T19:11:35.878644+0000","last_peered":"2024-04-25T19:11:35.878644+0000","last_clean":"2024-04-25T19:11:35.878644+0000","last_became_active":"2024-04-25T19:11:29.934928+0000","last_became_peered":"2024-04-25T19:11:29.934928+0000","last_unstale":"2024-04-25T19:11:35.878644+0000","last_undegraded":"2024-04-25T19:11:35.878644+0000","last_fullsized":"2024-04-25T19:11:35.878644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963243+0000","last_change":"2024-04-25T19:11:28.940436+0000","last_active":"2024-04-25T19:11:34.963243+0000","last_peered":"2024-04-25T19:11:34.963243+0000","last_clean":"2024-04-25T19:11:34.963243+0000","last_became_active":"2024-04-25T19:11:28.940050+0000","last_became_peered":"2024-04-25T19:11:28.940050+0000","last_unstale":"2024-04-25T19:11:34.963243+0000","last_undegraded":"2024-04-25T19:11:34.963243+0000","last_fullsized":"2024-04-25T19:11:34.963243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989678+0000","last_change":"2024-04-25T19:11:34.115813+0000","last_active":"2024-04-25T19:11:36.989678+0000","last_peered":"2024-04-25T19:11:36.989678+0000","last_clean":"2024-04-25T19:11:36.989678+0000","last_became_active":"2024-04-25T19:11:34.115268+0000","last_became_peered":"2024-04-25T19:11:34.115268+0000","last_unstale":"2024-04-25T19:11:36.989678+0000","last_undegraded":"2024-04-25T19:11:36.989678+0000","last_fullsized":"2024-04-25T19:11:36.989678+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:54.194127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171971+0000","last_change":"2024-04-25T19:11:29.945249+0000","last_active":"2024-04-25T19:11:37.171971+0000","last_peered":"2024-04-25T19:11:37.171971+0000","last_clean":"2024-04-25T19:11:37.171971+0000","last_became_active":"2024-04-25T19:11:29.944930+0000","last_became_peered":"2024-04-25T19:11:29.944930+0000","last_unstale":"2024-04-25T19:11:37.171971+0000","last_undegraded":"2024-04-25T19:11:37.171971+0000","last_fullsized":"2024-04-25T19:11:37.171971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470980+0000","last_change":"2024-04-25T19:11:28.937963+0000","last_active":"2024-04-25T19:11:35.470980+0000","last_peered":"2024-04-25T19:11:35.470980+0000","last_clean":"2024-04-25T19:11:35.470980+0000","last_became_active":"2024-04-25T19:11:28.936818+0000","last_became_peered":"2024-04-25T19:11:28.936818+0000","last_unstale":"2024-04-25T19:11:35.470980+0000","last_undegraded":"2024-04-25T19:11:35.470980+0000","last_fullsized":"2024-04-25T19:11:35.470980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170969+0000","last_change":"2024-04-25T19:11:34.113848+0000","last_active":"2024-04-25T19:11:37.170969+0000","last_peered":"2024-04-25T19:11:37.170969+0000","last_clean":"2024-04-25T19:11:37.170969+0000","last_became_active":"2024-04-25T19:11:34.113403+0000","last_became_peered":"2024-04-25T19:11:34.113403+0000","last_unstale":"2024-04-25T19:11:37.170969+0000","last_undegraded":"2024-04-25T19:11:37.170969+0000","last_fullsized":"2024-04-25T19:11:37.170969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:43.702771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470714+0000","last_change":"2024-04-25T19:11:29.930348+0000","last_active":"2024-04-25T19:11:35.470714+0000","last_peered":"2024-04-25T19:11:35.470714+0000","last_clean":"2024-04-25T19:11:35.470714+0000","last_became_active":"2024-04-25T19:11:29.930028+0000","last_became_peered":"2024-04-25T19:11:29.930028+0000","last_unstale":"2024-04-25T19:11:35.470714+0000","last_undegraded":"2024-04-25T19:11:35.470714+0000","last_fullsized":"2024-04-25T19:11:35.470714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170925+0000","last_change":"2024-04-25T19:11:28.939508+0000","last_active":"2024-04-25T19:11:37.170925+0000","last_peered":"2024-04-25T19:11:37.170925+0000","last_clean":"2024-04-25T19:11:37.170925+0000","last_became_active":"2024-04-25T19:11:28.939050+0000","last_became_peered":"2024-04-25T19:11:28.939050+0000","last_unstale":"2024-04-25T19:11:37.170925+0000","last_undegraded":"2024-04-25T19:11:37.170925+0000","last_fullsized":"2024-04-25T19:11:37.170925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:00.743906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878165+0000","last_change":"2024-04-25T19:11:34.110317+0000","last_active":"2024-04-25T19:11:35.878165+0000","last_peered":"2024-04-25T19:11:35.878165+0000","last_clean":"2024-04-25T19:11:35.878165+0000","last_became_active":"2024-04-25T19:11:34.109906+0000","last_became_peered":"2024-04-25T19:11:34.109906+0000","last_unstale":"2024-04-25T19:11:35.878165+0000","last_undegraded":"2024-04-25T19:11:35.878165+0000","last_fullsized":"2024-04-25T19:11:35.878165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:35.687485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962541+0000","last_change":"2024-04-25T19:11:29.952079+0000","last_active":"2024-04-25T19:11:34.962541+0000","last_peered":"2024-04-25T19:11:34.962541+0000","last_clean":"2024-04-25T19:11:34.962541+0000","last_became_active":"2024-04-25T19:11:29.951437+0000","last_became_peered":"2024-04-25T19:11:29.951437+0000","last_unstale":"2024-04-25T19:11:34.962541+0000","last_undegraded":"2024-04-25T19:11:34.962541+0000","last_fullsized":"2024-04-25T19:11:34.962541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470828+0000","last_change":"2024-04-25T19:11:28.933142+0000","last_active":"2024-04-25T19:11:35.470828+0000","last_peered":"2024-04-25T19:11:35.470828+0000","last_clean":"2024-04-25T19:11:35.470828+0000","last_became_active":"2024-04-25T19:11:28.932682+0000","last_became_peered":"2024-04-25T19:11:28.932682+0000","last_unstale":"2024-04-25T19:11:35.470828+0000","last_undegraded":"2024-04-25T19:11:35.470828+0000","last_fullsized":"2024-04-25T19:11:35.470828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:38.236609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470450+0000","last_change":"2024-04-25T19:11:33.971859+0000","last_active":"2024-04-25T19:11:35.470450+0000","last_peered":"2024-04-25T19:11:35.470450+0000","last_clean":"2024-04-25T19:11:35.470450+0000","last_became_active":"2024-04-25T19:11:33.971349+0000","last_became_peered":"2024-04-25T19:11:33.971349+0000","last_unstale":"2024-04-25T19:11:35.470450+0000","last_undegraded":"2024-04-25T19:11:35.470450+0000","last_fullsized":"2024-04-25T19:11:35.470450+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:43.916700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990771+0000","last_change":"2024-04-25T19:11:29.929623+0000","last_active":"2024-04-25T19:11:36.990771+0000","last_peered":"2024-04-25T19:11:36.990771+0000","last_clean":"2024-04-25T19:11:36.990771+0000","last_became_active":"2024-04-25T19:11:29.929306+0000","last_became_peered":"2024-04-25T19:11:29.929306+0000","last_unstale":"2024-04-25T19:11:36.990771+0000","last_undegraded":"2024-04-25T19:11:36.990771+0000","last_fullsized":"2024-04-25T19:11:36.990771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:49.958865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963468+0000","last_change":"2024-04-25T19:11:28.933056+0000","last_active":"2024-04-25T19:11:34.963468+0000","last_peered":"2024-04-25T19:11:34.963468+0000","last_clean":"2024-04-25T19:11:34.963468+0000","last_became_active":"2024-04-25T19:11:28.930233+0000","last_became_peered":"2024-04-25T19:11:28.930233+0000","last_unstale":"2024-04-25T19:11:34.963468+0000","last_undegraded":"2024-04-25T19:11:34.963468+0000","last_fullsized":"2024-04-25T19:11:34.963468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470757+0000","last_change":"2024-04-25T19:11:33.972447+0000","last_active":"2024-04-25T19:11:35.470757+0000","last_peered":"2024-04-25T19:11:35.470757+0000","last_clean":"2024-04-25T19:11:35.470757+0000","last_became_active":"2024-04-25T19:11:33.970645+0000","last_became_peered":"2024-04-25T19:11:33.970645+0000","last_unstale":"2024-04-25T19:11:35.470757+0000","last_undegraded":"2024-04-25T19:11:35.470757+0000","last_fullsized":"2024-04-25T19:11:35.470757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:40.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470256+0000","last_change":"2024-04-25T19:11:29.930967+0000","last_active":"2024-04-25T19:11:35.470256+0000","last_peered":"2024-04-25T19:11:35.470256+0000","last_clean":"2024-04-25T19:11:35.470256+0000","last_became_active":"2024-04-25T19:11:29.930682+0000","last_became_peered":"2024-04-25T19:11:29.930682+0000","last_unstale":"2024-04-25T19:11:35.470256+0000","last_undegraded":"2024-04-25T19:11:35.470256+0000","last_fullsized":"2024-04-25T19:11:35.470256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960304+0000","last_change":"2024-04-25T19:11:28.932205+0000","last_active":"2024-04-25T19:11:34.960304+0000","last_peered":"2024-04-25T19:11:34.960304+0000","last_clean":"2024-04-25T19:11:34.960304+0000","last_became_active":"2024-04-25T19:11:28.931851+0000","last_became_peered":"2024-04-25T19:11:28.931851+0000","last_unstale":"2024-04-25T19:11:34.960304+0000","last_undegraded":"2024-04-25T19:11:34.960304+0000","last_fullsized":"2024-04-25T19:11:34.960304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.471128+0000","last_change":"2024-04-25T19:11:33.971485+0000","last_active":"2024-04-25T19:11:35.471128+0000","last_peered":"2024-04-25T19:11:35.471128+0000","last_clean":"2024-04-25T19:11:35.471128+0000","last_became_active":"2024-04-25T19:11:33.971022+0000","last_became_peered":"2024-04-25T19:11:33.971022+0000","last_unstale":"2024-04-25T19:11:35.471128+0000","last_undegraded":"2024-04-25T19:11:35.471128+0000","last_fullsized":"2024-04-25T19:11:35.471128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:38.690554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878004+0000","last_change":"2024-04-25T19:11:29.936287+0000","last_active":"2024-04-25T19:11:35.878004+0000","last_peered":"2024-04-25T19:11:35.878004+0000","last_clean":"2024-04-25T19:11:35.878004+0000","last_became_active":"2024-04-25T19:11:29.935945+0000","last_became_peered":"2024-04-25T19:11:29.935945+0000","last_unstale":"2024-04-25T19:11:35.878004+0000","last_undegraded":"2024-04-25T19:11:35.878004+0000","last_fullsized":"2024-04-25T19:11:35.878004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:41.846903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170920+0000","last_change":"2024-04-25T19:11:28.940966+0000","last_active":"2024-04-25T19:11:37.170920+0000","last_peered":"2024-04-25T19:11:37.170920+0000","last_clean":"2024-04-25T19:11:37.170920+0000","last_became_active":"2024-04-25T19:11:28.939773+0000","last_became_peered":"2024-04-25T19:11:28.939773+0000","last_unstale":"2024-04-25T19:11:37.170920+0000","last_undegraded":"2024-04-25T19:11:37.170920+0000","last_fullsized":"2024-04-25T19:11:37.170920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878537+0000","last_change":"2024-04-25T19:11:34.113282+0000","last_active":"2024-04-25T19:11:35.878537+0000","last_peered":"2024-04-25T19:11:35.878537+0000","last_clean":"2024-04-25T19:11:35.878537+0000","last_became_active":"2024-04-25T19:11:34.112444+0000","last_became_peered":"2024-04-25T19:11:34.112444+0000","last_unstale":"2024-04-25T19:11:35.878537+0000","last_undegraded":"2024-04-25T19:11:35.878537+0000","last_fullsized":"2024-04-25T19:11:35.878537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:43.702771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170832+0000","last_change":"2024-04-25T19:11:29.949528+0000","last_active":"2024-04-25T19:11:37.170832+0000","last_peered":"2024-04-25T19:11:37.170832+0000","last_clean":"2024-04-25T19:11:37.170832+0000","last_became_active":"2024-04-25T19:11:29.948760+0000","last_became_peered":"2024-04-25T19:11:29.948760+0000","last_unstale":"2024-04-25T19:11:37.170832+0000","last_undegraded":"2024-04-25T19:11:37.170832+0000","last_fullsized":"2024-04-25T19:11:37.170832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963446+0000","last_change":"2024-04-25T19:11:28.928976+0000","last_active":"2024-04-25T19:11:34.963446+0000","last_peered":"2024-04-25T19:11:34.963446+0000","last_clean":"2024-04-25T19:11:34.963446+0000","last_became_active":"2024-04-25T19:11:28.928563+0000","last_became_peered":"2024-04-25T19:11:28.928563+0000","last_unstale":"2024-04-25T19:11:34.963446+0000","last_undegraded":"2024-04-25T19:11:34.963446+0000","last_fullsized":"2024-04-25T19:11:34.963446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":117,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171444+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:11:37.171444+0000","last_peered":"2024-04-25T19:11:37.171444+0000","last_clean":"2024-04-25T19:11:37.171444+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:11:37.171444+0000","last_undegraded":"2024-04-25T19:11:37.171444+0000","last_fullsized":"2024-04-25T19:11:37.171444+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962407+0000","last_change":"2024-04-25T19:11:34.113394+0000","last_active":"2024-04-25T19:11:34.962407+0000","last_peered":"2024-04-25T19:11:34.962407+0000","last_clean":"2024-04-25T19:11:34.962407+0000","last_became_active":"2024-04-25T19:11:34.112950+0000","last_became_peered":"2024-04-25T19:11:34.112950+0000","last_unstale":"2024-04-25T19:11:34.962407+0000","last_undegraded":"2024-04-25T19:11:34.962407+0000","last_fullsized":"2024-04-25T19:11:34.962407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:43.916700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171981+0000","last_change":"2024-04-25T19:11:29.946818+0000","last_active":"2024-04-25T19:11:37.171981+0000","last_peered":"2024-04-25T19:11:37.171981+0000","last_clean":"2024-04-25T19:11:37.171981+0000","last_became_active":"2024-04-25T19:11:29.946517+0000","last_became_peered":"2024-04-25T19:11:29.946517+0000","last_unstale":"2024-04-25T19:11:37.171981+0000","last_undegraded":"2024-04-25T19:11:37.171981+0000","last_fullsized":"2024-04-25T19:11:37.171981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960736+0000","last_change":"2024-04-25T19:11:28.928753+0000","last_active":"2024-04-25T19:11:34.960736+0000","last_peered":"2024-04-25T19:11:34.960736+0000","last_clean":"2024-04-25T19:11:34.960736+0000","last_became_active":"2024-04-25T19:11:28.928388+0000","last_became_peered":"2024-04-25T19:11:28.928388+0000","last_unstale":"2024-04-25T19:11:34.960736+0000","last_undegraded":"2024-04-25T19:11:34.960736+0000","last_fullsized":"2024-04-25T19:11:34.960736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961773+0000","last_change":"2024-04-25T19:11:34.110545+0000","last_active":"2024-04-25T19:11:34.961773+0000","last_peered":"2024-04-25T19:11:34.961773+0000","last_clean":"2024-04-25T19:11:34.961773+0000","last_became_active":"2024-04-25T19:11:34.110024+0000","last_became_peered":"2024-04-25T19:11:34.110024+0000","last_unstale":"2024-04-25T19:11:34.961773+0000","last_undegraded":"2024-04-25T19:11:34.961773+0000","last_fullsized":"2024-04-25T19:11:34.961773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:46.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962527+0000","last_change":"2024-04-25T19:11:29.950030+0000","last_active":"2024-04-25T19:11:34.962527+0000","last_peered":"2024-04-25T19:11:34.962527+0000","last_clean":"2024-04-25T19:11:34.962527+0000","last_became_active":"2024-04-25T19:11:29.949424+0000","last_became_peered":"2024-04-25T19:11:29.949424+0000","last_unstale":"2024-04-25T19:11:34.962527+0000","last_undegraded":"2024-04-25T19:11:34.962527+0000","last_fullsized":"2024-04-25T19:11:34.962527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:48.917939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470828+0000","last_change":"2024-04-25T19:11:28.934208+0000","last_active":"2024-04-25T19:11:35.470828+0000","last_peered":"2024-04-25T19:11:35.470828+0000","last_clean":"2024-04-25T19:11:35.470828+0000","last_became_active":"2024-04-25T19:11:28.933866+0000","last_became_peered":"2024-04-25T19:11:28.933866+0000","last_unstale":"2024-04-25T19:11:35.470828+0000","last_undegraded":"2024-04-25T19:11:35.470828+0000","last_fullsized":"2024-04-25T19:11:35.470828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:30.900062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878372+0000","last_change":"2024-04-25T19:11:33.980318+0000","last_active":"2024-04-25T19:11:35.878372+0000","last_peered":"2024-04-25T19:11:35.878372+0000","last_clean":"2024-04-25T19:11:35.878372+0000","last_became_active":"2024-04-25T19:11:33.979818+0000","last_became_peered":"2024-04-25T19:11:33.979818+0000","last_unstale":"2024-04-25T19:11:35.878372+0000","last_undegraded":"2024-04-25T19:11:35.878372+0000","last_fullsized":"2024-04-25T19:11:35.878372+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:38.690554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172468+0000","last_change":"2024-04-25T19:11:29.940479+0000","last_active":"2024-04-25T19:11:37.172468+0000","last_peered":"2024-04-25T19:11:37.172468+0000","last_clean":"2024-04-25T19:11:37.172468+0000","last_became_active":"2024-04-25T19:11:29.940140+0000","last_became_peered":"2024-04-25T19:11:29.940140+0000","last_unstale":"2024-04-25T19:11:37.172468+0000","last_undegraded":"2024-04-25T19:11:37.172468+0000","last_fullsized":"2024-04-25T19:11:37.172468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:56.761797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960699+0000","last_change":"2024-04-25T19:11:28.933352+0000","last_active":"2024-04-25T19:11:34.960699+0000","last_peered":"2024-04-25T19:11:34.960699+0000","last_clean":"2024-04-25T19:11:34.960699+0000","last_became_active":"2024-04-25T19:11:28.933016+0000","last_became_peered":"2024-04-25T19:11:28.933016+0000","last_unstale":"2024-04-25T19:11:34.960699+0000","last_undegraded":"2024-04-25T19:11:34.960699+0000","last_fullsized":"2024-04-25T19:11:34.960699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961463+0000","last_change":"2024-04-25T19:11:34.110567+0000","last_active":"2024-04-25T19:11:34.961463+0000","last_peered":"2024-04-25T19:11:34.961463+0000","last_clean":"2024-04-25T19:11:34.961463+0000","last_became_active":"2024-04-25T19:11:34.110074+0000","last_became_peered":"2024-04-25T19:11:34.110074+0000","last_unstale":"2024-04-25T19:11:34.961463+0000","last_undegraded":"2024-04-25T19:11:34.961463+0000","last_fullsized":"2024-04-25T19:11:34.961463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:54.738252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172199+0000","last_change":"2024-04-25T19:11:29.939683+0000","last_active":"2024-04-25T19:11:37.172199+0000","last_peered":"2024-04-25T19:11:37.172199+0000","last_clean":"2024-04-25T19:11:37.172199+0000","last_became_active":"2024-04-25T19:11:29.939372+0000","last_became_peered":"2024-04-25T19:11:29.939372+0000","last_unstale":"2024-04-25T19:11:37.172199+0000","last_undegraded":"2024-04-25T19:11:37.172199+0000","last_fullsized":"2024-04-25T19:11:37.172199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:51.175553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989410+0000","last_change":"2024-04-25T19:11:28.928627+0000","last_active":"2024-04-25T19:11:36.989410+0000","last_peered":"2024-04-25T19:11:36.989410+0000","last_clean":"2024-04-25T19:11:36.989410+0000","last_became_active":"2024-04-25T19:11:28.928016+0000","last_became_peered":"2024-04-25T19:11:28.928016+0000","last_unstale":"2024-04-25T19:11:36.989410+0000","last_undegraded":"2024-04-25T19:11:36.989410+0000","last_fullsized":"2024-04-25T19:11:36.989410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:16.833337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989757+0000","last_change":"2024-04-25T19:11:33.971762+0000","last_active":"2024-04-25T19:11:36.989757+0000","last_peered":"2024-04-25T19:11:36.989757+0000","last_clean":"2024-04-25T19:11:36.989757+0000","last_became_active":"2024-04-25T19:11:33.971317+0000","last_became_peered":"2024-04-25T19:11:33.971317+0000","last_unstale":"2024-04-25T19:11:36.989757+0000","last_undegraded":"2024-04-25T19:11:36.989757+0000","last_fullsized":"2024-04-25T19:11:36.989757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:23.748216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171561+0000","last_change":"2024-04-25T19:11:29.938906+0000","last_active":"2024-04-25T19:11:37.171561+0000","last_peered":"2024-04-25T19:11:37.171561+0000","last_clean":"2024-04-25T19:11:37.171561+0000","last_became_active":"2024-04-25T19:11:29.938575+0000","last_became_peered":"2024-04-25T19:11:29.938575+0000","last_unstale":"2024-04-25T19:11:37.171561+0000","last_undegraded":"2024-04-25T19:11:37.171561+0000","last_fullsized":"2024-04-25T19:11:37.171561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960132+0000","last_change":"2024-04-25T19:11:28.931773+0000","last_active":"2024-04-25T19:11:34.960132+0000","last_peered":"2024-04-25T19:11:34.960132+0000","last_clean":"2024-04-25T19:11:34.960132+0000","last_became_active":"2024-04-25T19:11:28.931409+0000","last_became_peered":"2024-04-25T19:11:28.931409+0000","last_unstale":"2024-04-25T19:11:34.960132+0000","last_undegraded":"2024-04-25T19:11:34.960132+0000","last_fullsized":"2024-04-25T19:11:34.960132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:27:30.904833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878361+0000","last_change":"2024-04-25T19:11:33.975482+0000","last_active":"2024-04-25T19:11:35.878361+0000","last_peered":"2024-04-25T19:11:35.878361+0000","last_clean":"2024-04-25T19:11:35.878361+0000","last_became_active":"2024-04-25T19:11:33.975042+0000","last_became_peered":"2024-04-25T19:11:33.975042+0000","last_unstale":"2024-04-25T19:11:35.878361+0000","last_undegraded":"2024-04-25T19:11:35.878361+0000","last_fullsized":"2024-04-25T19:11:35.878361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.795134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961644+0000","last_change":"2024-04-25T19:11:29.943783+0000","last_active":"2024-04-25T19:11:34.961644+0000","last_peered":"2024-04-25T19:11:34.961644+0000","last_clean":"2024-04-25T19:11:34.961644+0000","last_became_active":"2024-04-25T19:11:29.942647+0000","last_became_peered":"2024-04-25T19:11:29.942647+0000","last_unstale":"2024-04-25T19:11:34.961644+0000","last_undegraded":"2024-04-25T19:11:34.961644+0000","last_fullsized":"2024-04-25T19:11:34.961644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961794+0000","last_change":"2024-04-25T19:11:28.934715+0000","last_active":"2024-04-25T19:11:34.961794+0000","last_peered":"2024-04-25T19:11:34.961794+0000","last_clean":"2024-04-25T19:11:34.961794+0000","last_became_active":"2024-04-25T19:11:28.934331+0000","last_became_peered":"2024-04-25T19:11:28.934331+0000","last_unstale":"2024-04-25T19:11:34.961794+0000","last_undegraded":"2024-04-25T19:11:34.961794+0000","last_fullsized":"2024-04-25T19:11:34.961794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171897+0000","last_change":"2024-04-25T19:11:34.115885+0000","last_active":"2024-04-25T19:11:37.171897+0000","last_peered":"2024-04-25T19:11:37.171897+0000","last_clean":"2024-04-25T19:11:37.171897+0000","last_became_active":"2024-04-25T19:11:34.115489+0000","last_became_peered":"2024-04-25T19:11:34.115489+0000","last_unstale":"2024-04-25T19:11:37.171897+0000","last_undegraded":"2024-04-25T19:11:37.171897+0000","last_fullsized":"2024-04-25T19:11:37.171897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:15.088779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171931+0000","last_change":"2024-04-25T19:11:29.937605+0000","last_active":"2024-04-25T19:11:37.171931+0000","last_peered":"2024-04-25T19:11:37.171931+0000","last_clean":"2024-04-25T19:11:37.171931+0000","last_became_active":"2024-04-25T19:11:29.937045+0000","last_became_peered":"2024-04-25T19:11:29.937045+0000","last_unstale":"2024-04-25T19:11:37.171931+0000","last_undegraded":"2024-04-25T19:11:37.171931+0000","last_fullsized":"2024-04-25T19:11:37.171931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:39.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963413+0000","last_change":"2024-04-25T19:11:28.937550+0000","last_active":"2024-04-25T19:11:34.963413+0000","last_peered":"2024-04-25T19:11:34.963413+0000","last_clean":"2024-04-25T19:11:34.963413+0000","last_became_active":"2024-04-25T19:11:28.937224+0000","last_became_peered":"2024-04-25T19:11:28.937224+0000","last_unstale":"2024-04-25T19:11:34.963413+0000","last_undegraded":"2024-04-25T19:11:34.963413+0000","last_fullsized":"2024-04-25T19:11:34.963413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171513+0000","last_change":"2024-04-25T19:11:34.112538+0000","last_active":"2024-04-25T19:11:37.171513+0000","last_peered":"2024-04-25T19:11:37.171513+0000","last_clean":"2024-04-25T19:11:37.171513+0000","last_became_active":"2024-04-25T19:11:34.112105+0000","last_became_peered":"2024-04-25T19:11:34.112105+0000","last_unstale":"2024-04-25T19:11:37.171513+0000","last_undegraded":"2024-04-25T19:11:37.171513+0000","last_fullsized":"2024-04-25T19:11:37.171513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:04.753913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878413+0000","last_change":"2024-04-25T19:11:29.940791+0000","last_active":"2024-04-25T19:11:35.878413+0000","last_peered":"2024-04-25T19:11:35.878413+0000","last_clean":"2024-04-25T19:11:35.878413+0000","last_became_active":"2024-04-25T19:11:29.940494+0000","last_became_peered":"2024-04-25T19:11:29.940494+0000","last_unstale":"2024-04-25T19:11:35.878413+0000","last_undegraded":"2024-04-25T19:11:35.878413+0000","last_fullsized":"2024-04-25T19:11:35.878413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878662+0000","last_change":"2024-04-25T19:11:28.931517+0000","last_active":"2024-04-25T19:11:35.878662+0000","last_peered":"2024-04-25T19:11:35.878662+0000","last_clean":"2024-04-25T19:11:35.878662+0000","last_became_active":"2024-04-25T19:11:28.930964+0000","last_became_peered":"2024-04-25T19:11:28.930964+0000","last_unstale":"2024-04-25T19:11:35.878662+0000","last_undegraded":"2024-04-25T19:11:35.878662+0000","last_fullsized":"2024-04-25T19:11:35.878662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:14.977696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962359+0000","last_change":"2024-04-25T19:11:34.113386+0000","last_active":"2024-04-25T19:11:34.962359+0000","last_peered":"2024-04-25T19:11:34.962359+0000","last_clean":"2024-04-25T19:11:34.962359+0000","last_became_active":"2024-04-25T19:11:34.112820+0000","last_became_peered":"2024-04-25T19:11:34.112820+0000","last_unstale":"2024-04-25T19:11:34.962359+0000","last_undegraded":"2024-04-25T19:11:34.962359+0000","last_fullsized":"2024-04-25T19:11:34.962359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:04.753913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171892+0000","last_change":"2024-04-25T19:11:29.933966+0000","last_active":"2024-04-25T19:11:37.171892+0000","last_peered":"2024-04-25T19:11:37.171892+0000","last_clean":"2024-04-25T19:11:37.171892+0000","last_became_active":"2024-04-25T19:11:29.933549+0000","last_became_peered":"2024-04-25T19:11:29.933549+0000","last_unstale":"2024-04-25T19:11:37.171892+0000","last_undegraded":"2024-04-25T19:11:37.171892+0000","last_fullsized":"2024-04-25T19:11:37.171892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:39.517432+0000","last_change":"2024-04-25T19:11:28.936547+0000","last_active":"2024-04-25T19:11:39.517432+0000","last_peered":"2024-04-25T19:11:39.517432+0000","last_clean":"2024-04-25T19:11:39.517432+0000","last_became_active":"2024-04-25T19:11:28.935816+0000","last_became_peered":"2024-04-25T19:11:28.935816+0000","last_unstale":"2024-04-25T19:11:39.517432+0000","last_undegraded":"2024-04-25T19:11:39.517432+0000","last_fullsized":"2024-04-25T19:11:39.517432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470655+0000","last_change":"2024-04-25T19:11:33.972776+0000","last_active":"2024-04-25T19:11:35.470655+0000","last_peered":"2024-04-25T19:11:35.470655+0000","last_clean":"2024-04-25T19:11:35.470655+0000","last_became_active":"2024-04-25T19:11:33.971757+0000","last_became_peered":"2024-04-25T19:11:33.971757+0000","last_unstale":"2024-04-25T19:11:35.470655+0000","last_undegraded":"2024-04-25T19:11:35.470655+0000","last_fullsized":"2024-04-25T19:11:35.470655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:35.687485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990580+0000","last_change":"2024-04-25T19:11:29.925691+0000","last_active":"2024-04-25T19:11:36.990580+0000","last_peered":"2024-04-25T19:11:36.990580+0000","last_clean":"2024-04-25T19:11:36.990580+0000","last_became_active":"2024-04-25T19:11:29.924990+0000","last_became_peered":"2024-04-25T19:11:29.924990+0000","last_unstale":"2024-04-25T19:11:36.990580+0000","last_undegraded":"2024-04-25T19:11:36.990580+0000","last_fullsized":"2024-04-25T19:11:36.990580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:39.242588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470697+0000","last_change":"2024-04-25T19:11:28.937077+0000","last_active":"2024-04-25T19:11:35.470697+0000","last_peered":"2024-04-25T19:11:35.470697+0000","last_clean":"2024-04-25T19:11:35.470697+0000","last_became_active":"2024-04-25T19:11:28.936181+0000","last_became_peered":"2024-04-25T19:11:28.936181+0000","last_unstale":"2024-04-25T19:11:35.470697+0000","last_undegraded":"2024-04-25T19:11:35.470697+0000","last_fullsized":"2024-04-25T19:11:35.470697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990233+0000","last_change":"2024-04-25T19:11:33.972375+0000","last_active":"2024-04-25T19:11:36.990233+0000","last_peered":"2024-04-25T19:11:36.990233+0000","last_clean":"2024-04-25T19:11:36.990233+0000","last_became_active":"2024-04-25T19:11:33.971792+0000","last_became_peered":"2024-04-25T19:11:33.971792+0000","last_unstale":"2024-04-25T19:11:36.990233+0000","last_undegraded":"2024-04-25T19:11:36.990233+0000","last_fullsized":"2024-04-25T19:11:36.990233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:13.742925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878422+0000","last_change":"2024-04-25T19:11:29.943424+0000","last_active":"2024-04-25T19:11:35.878422+0000","last_peered":"2024-04-25T19:11:35.878422+0000","last_clean":"2024-04-25T19:11:35.878422+0000","last_became_active":"2024-04-25T19:11:29.943096+0000","last_became_peered":"2024-04-25T19:11:29.943096+0000","last_unstale":"2024-04-25T19:11:35.878422+0000","last_undegraded":"2024-04-25T19:11:35.878422+0000","last_fullsized":"2024-04-25T19:11:35.878422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962894+0000","last_change":"2024-04-25T19:11:28.934405+0000","last_active":"2024-04-25T19:11:34.962894+0000","last_peered":"2024-04-25T19:11:34.962894+0000","last_clean":"2024-04-25T19:11:34.962894+0000","last_became_active":"2024-04-25T19:11:28.933743+0000","last_became_peered":"2024-04-25T19:11:28.933743+0000","last_unstale":"2024-04-25T19:11:34.962894+0000","last_undegraded":"2024-04-25T19:11:34.962894+0000","last_fullsized":"2024-04-25T19:11:34.962894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171201+0000","last_change":"2024-04-25T19:11:33.969689+0000","last_active":"2024-04-25T19:11:37.171201+0000","last_peered":"2024-04-25T19:11:37.171201+0000","last_clean":"2024-04-25T19:11:37.171201+0000","last_became_active":"2024-04-25T19:11:33.969257+0000","last_became_peered":"2024-04-25T19:11:33.969257+0000","last_unstale":"2024-04-25T19:11:37.171201+0000","last_undegraded":"2024-04-25T19:11:37.171201+0000","last_fullsized":"2024-04-25T19:11:37.171201+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:57.736550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962757+0000","last_change":"2024-04-25T19:11:29.949944+0000","last_active":"2024-04-25T19:11:34.962757+0000","last_peered":"2024-04-25T19:11:34.962757+0000","last_clean":"2024-04-25T19:11:34.962757+0000","last_became_active":"2024-04-25T19:11:29.949521+0000","last_became_peered":"2024-04-25T19:11:29.949521+0000","last_unstale":"2024-04-25T19:11:34.962757+0000","last_undegraded":"2024-04-25T19:11:34.962757+0000","last_fullsized":"2024-04-25T19:11:34.962757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963008+0000","last_change":"2024-04-25T19:11:28.942291+0000","last_active":"2024-04-25T19:11:34.963008+0000","last_peered":"2024-04-25T19:11:34.963008+0000","last_clean":"2024-04-25T19:11:34.963008+0000","last_became_active":"2024-04-25T19:11:28.941951+0000","last_became_peered":"2024-04-25T19:11:28.941951+0000","last_unstale":"2024-04-25T19:11:34.963008+0000","last_undegraded":"2024-04-25T19:11:34.963008+0000","last_fullsized":"2024-04-25T19:11:34.963008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171184+0000","last_change":"2024-04-25T19:11:34.114829+0000","last_active":"2024-04-25T19:11:37.171184+0000","last_peered":"2024-04-25T19:11:37.171184+0000","last_clean":"2024-04-25T19:11:37.171184+0000","last_became_active":"2024-04-25T19:11:34.114452+0000","last_became_peered":"2024-04-25T19:11:34.114452+0000","last_unstale":"2024-04-25T19:11:37.171184+0000","last_undegraded":"2024-04-25T19:11:37.171184+0000","last_fullsized":"2024-04-25T19:11:37.171184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:54.738252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878339+0000","last_change":"2024-04-25T19:11:34.114794+0000","last_active":"2024-04-25T19:11:35.878339+0000","last_peered":"2024-04-25T19:11:35.878339+0000","last_clean":"2024-04-25T19:11:35.878339+0000","last_became_active":"2024-04-25T19:11:34.114376+0000","last_became_peered":"2024-04-25T19:11:34.114376+0000","last_unstale":"2024-04-25T19:11:35.878339+0000","last_undegraded":"2024-04-25T19:11:35.878339+0000","last_fullsized":"2024-04-25T19:11:35.878339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:40.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470153+0000","last_change":"2024-04-25T19:11:29.935850+0000","last_active":"2024-04-25T19:11:35.470153+0000","last_peered":"2024-04-25T19:11:35.470153+0000","last_clean":"2024-04-25T19:11:35.470153+0000","last_became_active":"2024-04-25T19:11:29.935546+0000","last_became_peered":"2024-04-25T19:11:29.935546+0000","last_unstale":"2024-04-25T19:11:35.470153+0000","last_undegraded":"2024-04-25T19:11:35.470153+0000","last_fullsized":"2024-04-25T19:11:35.470153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961554+0000","last_change":"2024-04-25T19:11:28.928204+0000","last_active":"2024-04-25T19:11:34.961554+0000","last_peered":"2024-04-25T19:11:34.961554+0000","last_clean":"2024-04-25T19:11:34.961554+0000","last_became_active":"2024-04-25T19:11:28.927812+0000","last_became_peered":"2024-04-25T19:11:28.927812+0000","last_unstale":"2024-04-25T19:11:34.961554+0000","last_undegraded":"2024-04-25T19:11:34.961554+0000","last_fullsized":"2024-04-25T19:11:34.961554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961976+0000","last_change":"2024-04-25T19:11:33.971937+0000","last_active":"2024-04-25T19:11:34.961976+0000","last_peered":"2024-04-25T19:11:34.961976+0000","last_clean":"2024-04-25T19:11:34.961976+0000","last_became_active":"2024-04-25T19:11:33.971297+0000","last_became_peered":"2024-04-25T19:11:33.971297+0000","last_unstale":"2024-04-25T19:11:34.961976+0000","last_undegraded":"2024-04-25T19:11:34.961976+0000","last_fullsized":"2024-04-25T19:11:34.961976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:52.951275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171272+0000","last_change":"2024-04-25T19:11:29.936690+0000","last_active":"2024-04-25T19:11:37.171272+0000","last_peered":"2024-04-25T19:11:37.171272+0000","last_clean":"2024-04-25T19:11:37.171272+0000","last_became_active":"2024-04-25T19:11:29.934968+0000","last_became_peered":"2024-04-25T19:11:29.934968+0000","last_unstale":"2024-04-25T19:11:37.171272+0000","last_undegraded":"2024-04-25T19:11:37.171272+0000","last_fullsized":"2024-04-25T19:11:37.171272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170564+0000","last_change":"2024-04-25T19:11:28.936664+0000","last_active":"2024-04-25T19:11:37.170564+0000","last_peered":"2024-04-25T19:11:37.170564+0000","last_clean":"2024-04-25T19:11:37.170564+0000","last_became_active":"2024-04-25T19:11:28.936236+0000","last_became_peered":"2024-04-25T19:11:28.936236+0000","last_unstale":"2024-04-25T19:11:37.170564+0000","last_undegraded":"2024-04-25T19:11:37.170564+0000","last_fullsized":"2024-04-25T19:11:37.170564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962069+0000","last_change":"2024-04-25T19:11:28.934644+0000","last_active":"2024-04-25T19:11:34.962069+0000","last_peered":"2024-04-25T19:11:34.962069+0000","last_clean":"2024-04-25T19:11:34.962069+0000","last_became_active":"2024-04-25T19:11:28.934282+0000","last_became_peered":"2024-04-25T19:11:28.934282+0000","last_unstale":"2024-04-25T19:11:34.962069+0000","last_undegraded":"2024-04-25T19:11:34.962069+0000","last_fullsized":"2024-04-25T19:11:34.962069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878411+0000","last_change":"2024-04-25T19:11:29.934419+0000","last_active":"2024-04-25T19:11:35.878411+0000","last_peered":"2024-04-25T19:11:35.878411+0000","last_clean":"2024-04-25T19:11:35.878411+0000","last_became_active":"2024-04-25T19:11:29.934119+0000","last_became_peered":"2024-04-25T19:11:29.934119+0000","last_unstale":"2024-04-25T19:11:35.878411+0000","last_undegraded":"2024-04-25T19:11:35.878411+0000","last_fullsized":"2024-04-25T19:11:35.878411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960537+0000","last_change":"2024-04-25T19:11:33.969976+0000","last_active":"2024-04-25T19:11:34.960537+0000","last_peered":"2024-04-25T19:11:34.960537+0000","last_clean":"2024-04-25T19:11:34.960537+0000","last_became_active":"2024-04-25T19:11:33.969493+0000","last_became_peered":"2024-04-25T19:11:33.969493+0000","last_unstale":"2024-04-25T19:11:34.960537+0000","last_undegraded":"2024-04-25T19:11:34.960537+0000","last_fullsized":"2024-04-25T19:11:34.960537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:45.880240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171502+0000","last_change":"2024-04-25T19:11:28.933487+0000","last_active":"2024-04-25T19:11:37.171502+0000","last_peered":"2024-04-25T19:11:37.171502+0000","last_clean":"2024-04-25T19:11:37.171502+0000","last_became_active":"2024-04-25T19:11:28.933112+0000","last_became_peered":"2024-04-25T19:11:28.933112+0000","last_unstale":"2024-04-25T19:11:37.171502+0000","last_undegraded":"2024-04-25T19:11:37.171502+0000","last_fullsized":"2024-04-25T19:11:37.171502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:44.780178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990049+0000","last_change":"2024-04-25T19:11:29.927210+0000","last_active":"2024-04-25T19:11:36.990049+0000","last_peered":"2024-04-25T19:11:36.990049+0000","last_clean":"2024-04-25T19:11:36.990049+0000","last_became_active":"2024-04-25T19:11:29.926143+0000","last_became_peered":"2024-04-25T19:11:29.926143+0000","last_unstale":"2024-04-25T19:11:36.990049+0000","last_undegraded":"2024-04-25T19:11:36.990049+0000","last_fullsized":"2024-04-25T19:11:36.990049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:31.906041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171522+0000","last_change":"2024-04-25T19:11:33.974927+0000","last_active":"2024-04-25T19:11:37.171522+0000","last_peered":"2024-04-25T19:11:37.171522+0000","last_clean":"2024-04-25T19:11:37.171522+0000","last_became_active":"2024-04-25T19:11:33.974526+0000","last_became_peered":"2024-04-25T19:11:33.974526+0000","last_unstale":"2024-04-25T19:11:37.171522+0000","last_undegraded":"2024-04-25T19:11:37.171522+0000","last_fullsized":"2024-04-25T19:11:37.171522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:32.367757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961363+0000","last_change":"2024-04-25T19:11:29.943955+0000","last_active":"2024-04-25T19:11:34.961363+0000","last_peered":"2024-04-25T19:11:34.961363+0000","last_clean":"2024-04-25T19:11:34.961363+0000","last_became_active":"2024-04-25T19:11:29.943620+0000","last_became_peered":"2024-04-25T19:11:29.943620+0000","last_unstale":"2024-04-25T19:11:34.961363+0000","last_undegraded":"2024-04-25T19:11:34.961363+0000","last_fullsized":"2024-04-25T19:11:34.961363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960143+0000","last_change":"2024-04-25T19:11:29.944224+0000","last_active":"2024-04-25T19:11:34.960143+0000","last_peered":"2024-04-25T19:11:34.960143+0000","last_clean":"2024-04-25T19:11:34.960143+0000","last_became_active":"2024-04-25T19:11:29.943851+0000","last_became_peered":"2024-04-25T19:11:29.943851+0000","last_unstale":"2024-04-25T19:11:34.960143+0000","last_undegraded":"2024-04-25T19:11:34.960143+0000","last_fullsized":"2024-04-25T19:11:34.960143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:31.906041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963271+0000","last_change":"2024-04-25T19:11:29.948559+0000","last_active":"2024-04-25T19:11:34.963271+0000","last_peered":"2024-04-25T19:11:34.963271+0000","last_clean":"2024-04-25T19:11:34.963271+0000","last_became_active":"2024-04-25T19:11:29.948261+0000","last_became_peered":"2024-04-25T19:11:29.948261+0000","last_unstale":"2024-04-25T19:11:34.963271+0000","last_undegraded":"2024-04-25T19:11:34.963271+0000","last_fullsized":"2024-04-25T19:11:34.963271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878297+0000","last_change":"2024-04-25T19:11:29.941924+0000","last_active":"2024-04-25T19:11:35.878297+0000","last_peered":"2024-04-25T19:11:35.878297+0000","last_clean":"2024-04-25T19:11:35.878297+0000","last_became_active":"2024-04-25T19:11:29.939317+0000","last_became_peered":"2024-04-25T19:11:29.939317+0000","last_unstale":"2024-04-25T19:11:35.878297+0000","last_undegraded":"2024-04-25T19:11:35.878297+0000","last_fullsized":"2024-04-25T19:11:35.878297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171676+0000","last_change":"2024-04-25T19:11:29.937716+0000","last_active":"2024-04-25T19:11:37.171676+0000","last_peered":"2024-04-25T19:11:37.171676+0000","last_clean":"2024-04-25T19:11:37.171676+0000","last_became_active":"2024-04-25T19:11:29.936902+0000","last_became_peered":"2024-04-25T19:11:29.936902+0000","last_unstale":"2024-04-25T19:11:37.171676+0000","last_undegraded":"2024-04-25T19:11:37.171676+0000","last_fullsized":"2024-04-25T19:11:37.171676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470475+0000","last_change":"2024-04-25T19:11:29.933948+0000","last_active":"2024-04-25T19:11:35.470475+0000","last_peered":"2024-04-25T19:11:35.470475+0000","last_clean":"2024-04-25T19:11:35.470475+0000","last_became_active":"2024-04-25T19:11:29.933609+0000","last_became_peered":"2024-04-25T19:11:29.933609+0000","last_unstale":"2024-04-25T19:11:35.470475+0000","last_undegraded":"2024-04-25T19:11:35.470475+0000","last_fullsized":"2024-04-25T19:11:35.470475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963143+0000","last_change":"2024-04-25T19:11:29.949696+0000","last_active":"2024-04-25T19:11:34.963143+0000","last_peered":"2024-04-25T19:11:34.963143+0000","last_clean":"2024-04-25T19:11:34.963143+0000","last_became_active":"2024-04-25T19:11:29.949244+0000","last_became_peered":"2024-04-25T19:11:29.949244+0000","last_unstale":"2024-04-25T19:11:34.963143+0000","last_undegraded":"2024-04-25T19:11:34.963143+0000","last_fullsized":"2024-04-25T19:11:34.963143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171872+0000","last_change":"2024-04-25T19:11:29.936507+0000","last_active":"2024-04-25T19:11:37.171872+0000","last_peered":"2024-04-25T19:11:37.171872+0000","last_clean":"2024-04-25T19:11:37.171872+0000","last_became_active":"2024-04-25T19:11:29.936194+0000","last_became_peered":"2024-04-25T19:11:29.936194+0000","last_unstale":"2024-04-25T19:11:37.171872+0000","last_undegraded":"2024-04-25T19:11:37.171872+0000","last_fullsized":"2024-04-25T19:11:37.171872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961467+0000","last_change":"2024-04-25T19:11:29.949296+0000","last_active":"2024-04-25T19:11:34.961467+0000","last_peered":"2024-04-25T19:11:34.961467+0000","last_clean":"2024-04-25T19:11:34.961467+0000","last_became_active":"2024-04-25T19:11:29.948939+0000","last_became_peered":"2024-04-25T19:11:29.948939+0000","last_unstale":"2024-04-25T19:11:34.961467+0000","last_undegraded":"2024-04-25T19:11:34.961467+0000","last_fullsized":"2024-04-25T19:11:34.961467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:39.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171201+0000","last_change":"2024-04-25T19:11:29.937245+0000","last_active":"2024-04-25T19:11:37.171201+0000","last_peered":"2024-04-25T19:11:37.171201+0000","last_clean":"2024-04-25T19:11:37.171201+0000","last_became_active":"2024-04-25T19:11:29.936924+0000","last_became_peered":"2024-04-25T19:11:29.936924+0000","last_unstale":"2024-04-25T19:11:37.171201+0000","last_undegraded":"2024-04-25T19:11:37.171201+0000","last_fullsized":"2024-04-25T19:11:37.171201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963148+0000","last_change":"2024-04-25T19:11:29.939024+0000","last_active":"2024-04-25T19:11:34.963148+0000","last_peered":"2024-04-25T19:11:34.963148+0000","last_clean":"2024-04-25T19:11:34.963148+0000","last_became_active":"2024-04-25T19:11:29.938702+0000","last_became_peered":"2024-04-25T19:11:29.938702+0000","last_unstale":"2024-04-25T19:11:34.963148+0000","last_undegraded":"2024-04-25T19:11:34.963148+0000","last_fullsized":"2024-04-25T19:11:34.963148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878155+0000","last_change":"2024-04-25T19:11:29.942883+0000","last_active":"2024-04-25T19:11:35.878155+0000","last_peered":"2024-04-25T19:11:35.878155+0000","last_clean":"2024-04-25T19:11:35.878155+0000","last_became_active":"2024-04-25T19:11:29.942371+0000","last_became_peered":"2024-04-25T19:11:29.942371+0000","last_unstale":"2024-04-25T19:11:35.878155+0000","last_undegraded":"2024-04-25T19:11:35.878155+0000","last_fullsized":"2024-04-25T19:11:35.878155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962242+0000","last_change":"2024-04-25T19:11:29.940287+0000","last_active":"2024-04-25T19:11:34.962242+0000","last_peered":"2024-04-25T19:11:34.962242+0000","last_clean":"2024-04-25T19:11:34.962242+0000","last_became_active":"2024-04-25T19:11:29.939973+0000","last_became_peered":"2024-04-25T19:11:29.939973+0000","last_unstale":"2024-04-25T19:11:34.962242+0000","last_undegraded":"2024-04-25T19:11:34.962242+0000","last_fullsized":"2024-04-25T19:11:34.962242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172189+0000","last_change":"2024-04-25T19:11:29.932331+0000","last_active":"2024-04-25T19:11:37.172189+0000","last_peered":"2024-04-25T19:11:37.172189+0000","last_clean":"2024-04-25T19:11:37.172189+0000","last_became_active":"2024-04-25T19:11:29.932030+0000","last_became_peered":"2024-04-25T19:11:29.932030+0000","last_unstale":"2024-04-25T19:11:37.172189+0000","last_undegraded":"2024-04-25T19:11:37.172189+0000","last_fullsized":"2024-04-25T19:11:37.172189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:48.917939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960501+0000","last_change":"2024-04-25T19:11:29.930099+0000","last_active":"2024-04-25T19:11:34.960501+0000","last_peered":"2024-04-25T19:11:34.960501+0000","last_clean":"2024-04-25T19:11:34.960501+0000","last_became_active":"2024-04-25T19:11:29.929729+0000","last_became_peered":"2024-04-25T19:11:29.929729+0000","last_unstale":"2024-04-25T19:11:34.960501+0000","last_undegraded":"2024-04-25T19:11:34.960501+0000","last_fullsized":"2024-04-25T19:11:34.960501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:39.242588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963790+0000","last_change":"2024-04-25T19:11:29.940716+0000","last_active":"2024-04-25T19:11:34.963790+0000","last_peered":"2024-04-25T19:11:34.963790+0000","last_clean":"2024-04-25T19:11:34.963790+0000","last_became_active":"2024-04-25T19:11:29.940412+0000","last_became_peered":"2024-04-25T19:11:29.940412+0000","last_unstale":"2024-04-25T19:11:34.963790+0000","last_undegraded":"2024-04-25T19:11:34.963790+0000","last_fullsized":"2024-04-25T19:11:34.963790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:35.993870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961354+0000","last_change":"2024-04-25T19:11:29.945896+0000","last_active":"2024-04-25T19:11:34.961354+0000","last_peered":"2024-04-25T19:11:34.961354+0000","last_clean":"2024-04-25T19:11:34.961354+0000","last_became_active":"2024-04-25T19:11:29.945590+0000","last_became_peered":"2024-04-25T19:11:29.945590+0000","last_unstale":"2024-04-25T19:11:34.961354+0000","last_undegraded":"2024-04-25T19:11:34.961354+0000","last_fullsized":"2024-04-25T19:11:34.961354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:42.722556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470247+0000","last_change":"2024-04-25T19:11:29.939162+0000","last_active":"2024-04-25T19:11:35.470247+0000","last_peered":"2024-04-25T19:11:35.470247+0000","last_clean":"2024-04-25T19:11:35.470247+0000","last_became_active":"2024-04-25T19:11:29.938821+0000","last_became_peered":"2024-04-25T19:11:29.938821+0000","last_unstale":"2024-04-25T19:11:35.470247+0000","last_undegraded":"2024-04-25T19:11:35.470247+0000","last_fullsized":"2024-04-25T19:11:35.470247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878300+0000","last_change":"2024-04-25T19:11:29.938034+0000","last_active":"2024-04-25T19:11:35.878300+0000","last_peered":"2024-04-25T19:11:35.878300+0000","last_clean":"2024-04-25T19:11:35.878300+0000","last_became_active":"2024-04-25T19:11:29.936771+0000","last_became_peered":"2024-04-25T19:11:29.936771+0000","last_unstale":"2024-04-25T19:11:35.878300+0000","last_undegraded":"2024-04-25T19:11:35.878300+0000","last_fullsized":"2024-04-25T19:11:35.878300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:23.833133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171095+0000","last_change":"2024-04-25T19:11:29.944382+0000","last_active":"2024-04-25T19:11:37.171095+0000","last_peered":"2024-04-25T19:11:37.171095+0000","last_clean":"2024-04-25T19:11:37.171095+0000","last_became_active":"2024-04-25T19:11:29.938702+0000","last_became_peered":"2024-04-25T19:11:29.938702+0000","last_unstale":"2024-04-25T19:11:37.171095+0000","last_undegraded":"2024-04-25T19:11:37.171095+0000","last_fullsized":"2024-04-25T19:11:37.171095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990488+0000","last_change":"2024-04-25T19:11:29.934918+0000","last_active":"2024-04-25T19:11:36.990488+0000","last_peered":"2024-04-25T19:11:36.990488+0000","last_clean":"2024-04-25T19:11:36.990488+0000","last_became_active":"2024-04-25T19:11:29.934600+0000","last_became_peered":"2024-04-25T19:11:29.934600+0000","last_unstale":"2024-04-25T19:11:36.990488+0000","last_undegraded":"2024-04-25T19:11:36.990488+0000","last_fullsized":"2024-04-25T19:11:36.990488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:15.027554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962499+0000","last_change":"2024-04-25T19:11:29.944854+0000","last_active":"2024-04-25T19:11:34.962499+0000","last_peered":"2024-04-25T19:11:34.962499+0000","last_clean":"2024-04-25T19:11:34.962499+0000","last_became_active":"2024-04-25T19:11:29.944528+0000","last_became_peered":"2024-04-25T19:11:29.944528+0000","last_unstale":"2024-04-25T19:11:34.962499+0000","last_undegraded":"2024-04-25T19:11:34.962499+0000","last_fullsized":"2024-04-25T19:11:34.962499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963024+0000","last_change":"2024-04-25T19:11:29.944061+0000","last_active":"2024-04-25T19:11:34.963024+0000","last_peered":"2024-04-25T19:11:34.963024+0000","last_clean":"2024-04-25T19:11:34.963024+0000","last_became_active":"2024-04-25T19:11:29.943745+0000","last_became_peered":"2024-04-25T19:11:29.943745+0000","last_unstale":"2024-04-25T19:11:34.963024+0000","last_undegraded":"2024-04-25T19:11:34.963024+0000","last_fullsized":"2024-04-25T19:11:34.963024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172433+0000","last_change":"2024-04-25T19:11:29.944585+0000","last_active":"2024-04-25T19:11:37.172433+0000","last_peered":"2024-04-25T19:11:37.172433+0000","last_clean":"2024-04-25T19:11:37.172433+0000","last_became_active":"2024-04-25T19:11:29.944112+0000","last_became_peered":"2024-04-25T19:11:29.944112+0000","last_unstale":"2024-04-25T19:11:37.172433+0000","last_undegraded":"2024-04-25T19:11:37.172433+0000","last_fullsized":"2024-04-25T19:11:37.172433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:31.654149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170915+0000","last_change":"2024-04-25T19:11:29.949991+0000","last_active":"2024-04-25T19:11:37.170915+0000","last_peered":"2024-04-25T19:11:37.170915+0000","last_clean":"2024-04-25T19:11:37.170915+0000","last_became_active":"2024-04-25T19:11:29.948567+0000","last_became_peered":"2024-04-25T19:11:29.948567+0000","last_unstale":"2024-04-25T19:11:37.170915+0000","last_undegraded":"2024-04-25T19:11:37.170915+0000","last_fullsized":"2024-04-25T19:11:37.170915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171414+0000","last_change":"2024-04-25T19:11:29.930321+0000","last_active":"2024-04-25T19:11:37.171414+0000","last_peered":"2024-04-25T19:11:37.171414+0000","last_clean":"2024-04-25T19:11:37.171414+0000","last_became_active":"2024-04-25T19:11:29.929930+0000","last_became_peered":"2024-04-25T19:11:29.929930+0000","last_unstale":"2024-04-25T19:11:37.171414+0000","last_undegraded":"2024-04-25T19:11:37.171414+0000","last_fullsized":"2024-04-25T19:11:37.171414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963030+0000","last_change":"2024-04-25T19:11:29.944694+0000","last_active":"2024-04-25T19:11:34.963030+0000","last_peered":"2024-04-25T19:11:34.963030+0000","last_clean":"2024-04-25T19:11:34.963030+0000","last_became_active":"2024-04-25T19:11:29.944055+0000","last_became_peered":"2024-04-25T19:11:29.944055+0000","last_unstale":"2024-04-25T19:11:34.963030+0000","last_undegraded":"2024-04-25T19:11:34.963030+0000","last_fullsized":"2024-04-25T19:11:34.963030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171775+0000","last_change":"2024-04-25T19:11:29.931815+0000","last_active":"2024-04-25T19:11:37.171775+0000","last_peered":"2024-04-25T19:11:37.171775+0000","last_clean":"2024-04-25T19:11:37.171775+0000","last_became_active":"2024-04-25T19:11:29.931523+0000","last_became_peered":"2024-04-25T19:11:29.931523+0000","last_unstale":"2024-04-25T19:11:37.171775+0000","last_undegraded":"2024-04-25T19:11:37.171775+0000","last_fullsized":"2024-04-25T19:11:37.171775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:42.722556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962270+0000","last_change":"2024-04-25T19:11:29.951038+0000","last_active":"2024-04-25T19:11:34.962270+0000","last_peered":"2024-04-25T19:11:34.962270+0000","last_clean":"2024-04-25T19:11:34.962270+0000","last_became_active":"2024-04-25T19:11:29.950615+0000","last_became_peered":"2024-04-25T19:11:29.950615+0000","last_unstale":"2024-04-25T19:11:34.962270+0000","last_undegraded":"2024-04-25T19:11:34.962270+0000","last_fullsized":"2024-04-25T19:11:34.962270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:31.654149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878604+0000","last_change":"2024-04-25T19:11:29.932658+0000","last_active":"2024-04-25T19:11:35.878604+0000","last_peered":"2024-04-25T19:11:35.878604+0000","last_clean":"2024-04-25T19:11:35.878604+0000","last_became_active":"2024-04-25T19:11:29.932270+0000","last_became_peered":"2024-04-25T19:11:29.932270+0000","last_unstale":"2024-04-25T19:11:35.878604+0000","last_undegraded":"2024-04-25T19:11:35.878604+0000","last_fullsized":"2024-04-25T19:11:35.878604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470928+0000","last_change":"2024-04-25T19:11:29.929765+0000","last_active":"2024-04-25T19:11:35.470928+0000","last_peered":"2024-04-25T19:11:35.470928+0000","last_clean":"2024-04-25T19:11:35.470928+0000","last_became_active":"2024-04-25T19:11:29.929452+0000","last_became_peered":"2024-04-25T19:11:29.929452+0000","last_unstale":"2024-04-25T19:11:35.470928+0000","last_undegraded":"2024-04-25T19:11:35.470928+0000","last_fullsized":"2024-04-25T19:11:35.470928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172357+0000","last_change":"2024-04-25T19:11:29.936916+0000","last_active":"2024-04-25T19:11:37.172357+0000","last_peered":"2024-04-25T19:11:37.172357+0000","last_clean":"2024-04-25T19:11:37.172357+0000","last_became_active":"2024-04-25T19:11:29.935057+0000","last_became_peered":"2024-04-25T19:11:29.935057+0000","last_unstale":"2024-04-25T19:11:37.172357+0000","last_undegraded":"2024-04-25T19:11:37.172357+0000","last_fullsized":"2024-04-25T19:11:37.172357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:35.993870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:11:53.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.210+0000 7fdf68d1f640 1 Processor -- start 2024-04-25T19:11:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.210+0000 7fdf68d1f640 1 -- start start 2024-04-25T19:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.211+0000 7fdf68d1f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf64161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.211+0000 7fdf68d1f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf64163e80 con 0x7fdf6406e5e0 2024-04-25T19:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.211+0000 7fdf68d1f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf64164160 con 0x7fdf640b5130 2024-04-25T19:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.211+0000 7fdf68d1f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fdf64163c30 con 0x7fdf640d4910 2024-04-25T19:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf61d9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdf6406e5e0 0x7fdf6415f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58888/0 (socket says 172.21.15.28:58888) 2024-04-25T19:11:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf61d9b640 1 -- 172.21.15.28:0/3148301772 learned_addr learned my addr 172.21.15.28:0/3148301772 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf62d9d640 1 --2- 172.21.15.28:0/3148301772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf64161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf6159a640 1 -- 172.21.15.28:0/3148301772 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3478088479 0 0) 0x7fdf64163e80 con 0x7fdf6406e5e0 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf6159a640 1 -- 172.21.15.28:0/3148301772 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf4c003520 con 0x7fdf6406e5e0 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf6159a640 1 -- 172.21.15.28:0/3148301772 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355125130 0 0) 0x7fdf64164160 con 0x7fdf640b5130 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf6159a640 1 -- 172.21.15.28:0/3148301772 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf64163e80 con 0x7fdf640b5130 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf62d9d640 1 -- 172.21.15.28:0/3148301772 >> v1:172.21.15.53:6789/0 conn(0x7fdf640b5130 legacy=0x7fdf6406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf62d9d640 1 -- 172.21.15.28:0/3148301772 >> v1:172.21.15.28:6789/0 conn(0x7fdf6406e5e0 legacy=0x7fdf6415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.212+0000 7fdf62d9d640 1 -- 172.21.15.28:0/3148301772 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64164440 con 0x7fdf640d4910 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.213+0000 7fdf62d9d640 1 --2- 172.21.15.28:0/3148301772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf64161130 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009dc0 tx=0x7fdf5c02a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=747a4d39c0e0c07d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.213+0000 7fdf6159a640 1 -- 172.21.15.28:0/3148301772 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c007740 con 0x7fdf640d4910 2024-04-25T19:11:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.213+0000 7fdf6159a640 1 -- 172.21.15.28:0/3148301772 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5c0078e0 con 0x7fdf640d4910 2024-04-25T19:11:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.213+0000 7fdf6159a640 1 -- 172.21.15.28:0/3148301772 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf5c007c90 con 0x7fdf640d4910 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.214+0000 7fdf68d1f640 1 -- 172.21.15.28:0/3148301772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 msgr2=0x7fdf64161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.214+0000 7fdf68d1f640 1 --2- 172.21.15.28:0/3148301772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf64161130 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdf5c009dc0 tx=0x7fdf5c02a6d0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.214+0000 7fdf68d1f640 1 -- 172.21.15.28:0/3148301772 shutdown_connections 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.214+0000 7fdf68d1f640 1 --2- 172.21.15.28:0/3148301772 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf64161130 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.214+0000 7fdf68d1f640 1 -- 172.21.15.28:0/3148301772 >> 172.21.15.28:0/3148301772 conn(0x7fdf640b7ad0 msgr2=0x7fdf640b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.214+0000 7fdf68d1f640 1 -- 172.21.15.28:0/3148301772 shutdown_connections 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.214+0000 7fdf68d1f640 1 -- 172.21.15.28:0/3148301772 wait complete. 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.216+0000 7fdf68d1f640 1 Processor -- start 2024-04-25T19:11:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.216+0000 7fdf68d1f640 1 -- start start 2024-04-25T19:11:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf68d1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf640b5130 0x7fdf64156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf68d1f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf6414ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf68d1f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf64155120 0x7fdf64158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf68d1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdf6415fee0 con 0x7fdf640b5130 2024-04-25T19:11:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf68d1f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdf64164a90 con 0x7fdf64155120 2024-04-25T19:11:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf68d1f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fdf64160060 con 0x7fdf640d4910 2024-04-25T19:11:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf6259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf640b5130 0x7fdf64156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf6259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf640b5130 0x7fdf64156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37136/0 (socket says 172.21.15.28:37136) 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf6259c640 1 -- 172.21.15.28:0/1620086291 learned_addr learned my addr 172.21.15.28:0/1620086291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.217+0000 7fdf62d9d640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf64155120 0x7fdf64158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf61d9b640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf6414ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf6259c640 1 -- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 msgr2=0x7fdf6414ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf6259c640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf6414ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf6259c640 1 -- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf64155120 msgr2=0x7fdf64158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf6259c640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf64155120 0x7fdf64158290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf6259c640 1 -- 172.21.15.28:0/1620086291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf64156b60 con 0x7fdf640b5130 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf61d9b640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf6414ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf62d9d640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf64155120 0x7fdf64158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf6259c640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf640b5130 0x7fdf64156260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdf54001eb0 tx=0x7fdf5400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf52ffd640 1 -- 172.21.15.28:0/1620086291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54017070 con 0x7fdf640b5130 2024-04-25T19:11:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf52ffd640 1 -- 172.21.15.28:0/1620086291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf540052d0 con 0x7fdf640b5130 2024-04-25T19:11:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf64161690 con 0x7fdf640b5130 2024-04-25T19:11:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf52ffd640 1 -- 172.21.15.28:0/1620086291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf54003810 con 0x7fdf640b5130 2024-04-25T19:11:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.218+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf64161c50 con 0x7fdf640b5130 2024-04-25T19:11:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.219+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf64050740 con 0x7fdf640b5130 2024-04-25T19:11:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.220+0000 7fdf52ffd640 1 -- 172.21.15.28:0/1620086291 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdf5401a070 con 0x7fdf640b5130 2024-04-25T19:11:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.220+0000 7fdf52ffd640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdf440744b0 0x7fdf440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.220+0000 7fdf52ffd640 1 -- 172.21.15.28:0/1620086291 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdf54094910 con 0x7fdf640b5130 2024-04-25T19:11:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.220+0000 7fdf61d9b640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdf440744b0 0x7fdf440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.221+0000 7fdf61d9b640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdf440744b0 0x7fdf440769c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdf58007fd0 tx=0x7fdf58009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.225+0000 7fdf52ffd640 1 -- 172.21.15.28:0/1620086291 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdf5405ed60 con 0x7fdf640b5130 2024-04-25T19:11:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.409+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdf64068e00 con 0x7fdf440744b0 2024-04-25T19:11:53.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.417+0000 7fdf52ffd640 1 -- 172.21.15.28:0/1620086291 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7fdf64068e00 con 0x7fdf440744b0 2024-04-25T19:11:53.423 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:53.425 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:11:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdf440744b0 msgr2=0x7fdf440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdf440744b0 0x7fdf440769c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdf58007fd0 tx=0x7fdf58009040 comp rx=0 tx=0).stop 2024-04-25T19:11:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf640b5130 msgr2=0x7fdf64156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf640b5130 0x7fdf64156260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdf54001eb0 tx=0x7fdf5400c820 comp rx=0 tx=0).stop 2024-04-25T19:11:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 shutdown_connections 2024-04-25T19:11:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdf440744b0 0x7fdf440769c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf64155120 0x7fdf64158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdf640d4910 0x7fdf6414ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 --2- 172.21.15.28:0/1620086291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf640b5130 0x7fdf64156260 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 >> 172.21.15.28:0/1620086291 conn(0x7fdf640b7ad0 msgr2=0x7fdf6415e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 shutdown_connections 2024-04-25T19:11:53.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.421+0000 7fdf68d1f640 1 -- 172.21.15.28:0/1620086291 wait complete. 2024-04-25T19:11:53.434 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-25T19:11:52.841746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252656,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722064,"statfs":{"total":773094113280,"available":772835393536,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003131"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878592+0000","last_change":"2024-04-25T19:11:28.938792+0000","last_active":"2024-04-25T19:11:35.878592+0000","last_peered":"2024-04-25T19:11:35.878592+0000","last_clean":"2024-04-25T19:11:35.878592+0000","last_became_active":"2024-04-25T19:11:28.938417+0000","last_became_peered":"2024-04-25T19:11:28.938417+0000","last_unstale":"2024-04-25T19:11:35.878592+0000","last_undegraded":"2024-04-25T19:11:35.878592+0000","last_fullsized":"2024-04-25T19:11:35.878592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:44.780178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171775+0000","last_change":"2024-04-25T19:11:29.937823+0000","last_active":"2024-04-25T19:11:37.171775+0000","last_peered":"2024-04-25T19:11:37.171775+0000","last_clean":"2024-04-25T19:11:37.171775+0000","last_became_active":"2024-04-25T19:11:29.937501+0000","last_became_peered":"2024-04-25T19:11:29.937501+0000","last_unstale":"2024-04-25T19:11:37.171775+0000","last_undegraded":"2024-04-25T19:11:37.171775+0000","last_fullsized":"2024-04-25T19:11:37.171775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170976+0000","last_change":"2024-04-25T19:11:34.113864+0000","last_active":"2024-04-25T19:11:37.170976+0000","last_peered":"2024-04-25T19:11:37.170976+0000","last_clean":"2024-04-25T19:11:37.170976+0000","last_became_active":"2024-04-25T19:11:34.113449+0000","last_became_peered":"2024-04-25T19:11:34.113449+0000","last_unstale":"2024-04-25T19:11:37.170976+0000","last_undegraded":"2024-04-25T19:11:37.170976+0000","last_fullsized":"2024-04-25T19:11:37.170976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.247905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170712+0000","last_change":"2024-04-25T19:11:28.934472+0000","last_active":"2024-04-25T19:11:37.170712+0000","last_peered":"2024-04-25T19:11:37.170712+0000","last_clean":"2024-04-25T19:11:37.170712+0000","last_became_active":"2024-04-25T19:11:28.934016+0000","last_became_peered":"2024-04-25T19:11:28.934016+0000","last_unstale":"2024-04-25T19:11:37.170712+0000","last_undegraded":"2024-04-25T19:11:37.170712+0000","last_fullsized":"2024-04-25T19:11:37.170712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:27:30.904833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878257+0000","last_change":"2024-04-25T19:11:29.944065+0000","last_active":"2024-04-25T19:11:35.878257+0000","last_peered":"2024-04-25T19:11:35.878257+0000","last_clean":"2024-04-25T19:11:35.878257+0000","last_became_active":"2024-04-25T19:11:29.943773+0000","last_became_peered":"2024-04-25T19:11:29.943773+0000","last_unstale":"2024-04-25T19:11:35.878257+0000","last_undegraded":"2024-04-25T19:11:35.878257+0000","last_fullsized":"2024-04-25T19:11:35.878257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:15.552217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989732+0000","last_change":"2024-04-25T19:11:33.975423+0000","last_active":"2024-04-25T19:11:36.989732+0000","last_peered":"2024-04-25T19:11:36.989732+0000","last_clean":"2024-04-25T19:11:36.989732+0000","last_became_active":"2024-04-25T19:11:33.974878+0000","last_became_peered":"2024-04-25T19:11:33.974878+0000","last_unstale":"2024-04-25T19:11:36.989732+0000","last_undegraded":"2024-04-25T19:11:36.989732+0000","last_fullsized":"2024-04-25T19:11:36.989732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:23.927981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170963+0000","last_change":"2024-04-25T19:11:28.937102+0000","last_active":"2024-04-25T19:11:37.170963+0000","last_peered":"2024-04-25T19:11:37.170963+0000","last_clean":"2024-04-25T19:11:37.170963+0000","last_became_active":"2024-04-25T19:11:28.936699+0000","last_became_peered":"2024-04-25T19:11:28.936699+0000","last_unstale":"2024-04-25T19:11:37.170963+0000","last_undegraded":"2024-04-25T19:11:37.170963+0000","last_fullsized":"2024-04-25T19:11:37.170963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:38.236609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962397+0000","last_change":"2024-04-25T19:11:29.947706+0000","last_active":"2024-04-25T19:11:34.962397+0000","last_peered":"2024-04-25T19:11:34.962397+0000","last_clean":"2024-04-25T19:11:34.962397+0000","last_became_active":"2024-04-25T19:11:29.945963+0000","last_became_peered":"2024-04-25T19:11:29.945963+0000","last_unstale":"2024-04-25T19:11:34.962397+0000","last_undegraded":"2024-04-25T19:11:34.962397+0000","last_fullsized":"2024-04-25T19:11:34.962397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:51.175553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990390+0000","last_change":"2024-04-25T19:11:33.975724+0000","last_active":"2024-04-25T19:11:36.990390+0000","last_peered":"2024-04-25T19:11:36.990390+0000","last_clean":"2024-04-25T19:11:36.990390+0000","last_became_active":"2024-04-25T19:11:33.974921+0000","last_became_peered":"2024-04-25T19:11:33.974921+0000","last_unstale":"2024-04-25T19:11:36.990390+0000","last_undegraded":"2024-04-25T19:11:36.990390+0000","last_fullsized":"2024-04-25T19:11:36.990390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:55.208890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170918+0000","last_change":"2024-04-25T19:11:28.940100+0000","last_active":"2024-04-25T19:11:37.170918+0000","last_peered":"2024-04-25T19:11:37.170918+0000","last_clean":"2024-04-25T19:11:37.170918+0000","last_became_active":"2024-04-25T19:11:28.939482+0000","last_became_peered":"2024-04-25T19:11:28.939482+0000","last_unstale":"2024-04-25T19:11:37.170918+0000","last_undegraded":"2024-04-25T19:11:37.170918+0000","last_fullsized":"2024-04-25T19:11:37.170918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962428+0000","last_change":"2024-04-25T19:11:29.945900+0000","last_active":"2024-04-25T19:11:34.962428+0000","last_peered":"2024-04-25T19:11:34.962428+0000","last_clean":"2024-04-25T19:11:34.962428+0000","last_became_active":"2024-04-25T19:11:29.945554+0000","last_became_peered":"2024-04-25T19:11:29.945554+0000","last_unstale":"2024-04-25T19:11:34.962428+0000","last_undegraded":"2024-04-25T19:11:34.962428+0000","last_fullsized":"2024-04-25T19:11:34.962428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990011+0000","last_change":"2024-04-25T19:11:33.975263+0000","last_active":"2024-04-25T19:11:36.990011+0000","last_peered":"2024-04-25T19:11:36.990011+0000","last_clean":"2024-04-25T19:11:36.990011+0000","last_became_active":"2024-04-25T19:11:33.974661+0000","last_became_peered":"2024-04-25T19:11:33.974661+0000","last_unstale":"2024-04-25T19:11:36.990011+0000","last_undegraded":"2024-04-25T19:11:36.990011+0000","last_fullsized":"2024-04-25T19:11:36.990011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:29.019166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961851+0000","last_change":"2024-04-25T19:11:29.939828+0000","last_active":"2024-04-25T19:11:34.961851+0000","last_peered":"2024-04-25T19:11:34.961851+0000","last_clean":"2024-04-25T19:11:34.961851+0000","last_became_active":"2024-04-25T19:11:29.939513+0000","last_became_peered":"2024-04-25T19:11:29.939513+0000","last_unstale":"2024-04-25T19:11:34.961851+0000","last_undegraded":"2024-04-25T19:11:34.961851+0000","last_fullsized":"2024-04-25T19:11:34.961851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171950+0000","last_change":"2024-04-25T19:11:28.937339+0000","last_active":"2024-04-25T19:11:37.171950+0000","last_peered":"2024-04-25T19:11:37.171950+0000","last_clean":"2024-04-25T19:11:37.171950+0000","last_became_active":"2024-04-25T19:11:28.936962+0000","last_became_peered":"2024-04-25T19:11:28.936962+0000","last_unstale":"2024-04-25T19:11:37.171950+0000","last_undegraded":"2024-04-25T19:11:37.171950+0000","last_fullsized":"2024-04-25T19:11:37.171950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:30.900062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990657+0000","last_change":"2024-04-25T19:11:33.975365+0000","last_active":"2024-04-25T19:11:36.990657+0000","last_peered":"2024-04-25T19:11:36.990657+0000","last_clean":"2024-04-25T19:11:36.990657+0000","last_became_active":"2024-04-25T19:11:33.974761+0000","last_became_peered":"2024-04-25T19:11:33.974761+0000","last_unstale":"2024-04-25T19:11:36.990657+0000","last_undegraded":"2024-04-25T19:11:36.990657+0000","last_fullsized":"2024-04-25T19:11:36.990657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:46.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171339+0000","last_change":"2024-04-25T19:11:29.935255+0000","last_active":"2024-04-25T19:11:37.171339+0000","last_peered":"2024-04-25T19:11:37.171339+0000","last_clean":"2024-04-25T19:11:37.171339+0000","last_became_active":"2024-04-25T19:11:29.934911+0000","last_became_peered":"2024-04-25T19:11:29.934911+0000","last_unstale":"2024-04-25T19:11:37.171339+0000","last_undegraded":"2024-04-25T19:11:37.171339+0000","last_fullsized":"2024-04-25T19:11:37.171339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470651+0000","last_change":"2024-04-25T19:11:28.937507+0000","last_active":"2024-04-25T19:11:35.470651+0000","last_peered":"2024-04-25T19:11:35.470651+0000","last_clean":"2024-04-25T19:11:35.470651+0000","last_became_active":"2024-04-25T19:11:28.936655+0000","last_became_peered":"2024-04-25T19:11:28.936655+0000","last_unstale":"2024-04-25T19:11:35.470651+0000","last_undegraded":"2024-04-25T19:11:35.470651+0000","last_fullsized":"2024-04-25T19:11:35.470651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990392+0000","last_change":"2024-04-25T19:11:33.974529+0000","last_active":"2024-04-25T19:11:36.990392+0000","last_peered":"2024-04-25T19:11:36.990392+0000","last_clean":"2024-04-25T19:11:36.990392+0000","last_became_active":"2024-04-25T19:11:33.973907+0000","last_became_peered":"2024-04-25T19:11:33.973907+0000","last_unstale":"2024-04-25T19:11:36.990392+0000","last_undegraded":"2024-04-25T19:11:36.990392+0000","last_fullsized":"2024-04-25T19:11:36.990392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:52.041722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171394+0000","last_change":"2024-04-25T19:11:29.930361+0000","last_active":"2024-04-25T19:11:37.171394+0000","last_peered":"2024-04-25T19:11:37.171394+0000","last_clean":"2024-04-25T19:11:37.171394+0000","last_became_active":"2024-04-25T19:11:29.930060+0000","last_became_peered":"2024-04-25T19:11:29.930060+0000","last_unstale":"2024-04-25T19:11:37.171394+0000","last_undegraded":"2024-04-25T19:11:37.171394+0000","last_fullsized":"2024-04-25T19:11:37.171394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470271+0000","last_change":"2024-04-25T19:11:28.930410+0000","last_active":"2024-04-25T19:11:35.470271+0000","last_peered":"2024-04-25T19:11:35.470271+0000","last_clean":"2024-04-25T19:11:35.470271+0000","last_became_active":"2024-04-25T19:11:28.930012+0000","last_became_peered":"2024-04-25T19:11:28.930012+0000","last_unstale":"2024-04-25T19:11:35.470271+0000","last_undegraded":"2024-04-25T19:11:35.470271+0000","last_fullsized":"2024-04-25T19:11:35.470271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.471102+0000","last_change":"2024-04-25T19:11:34.115931+0000","last_active":"2024-04-25T19:11:35.471102+0000","last_peered":"2024-04-25T19:11:35.471102+0000","last_clean":"2024-04-25T19:11:35.471102+0000","last_became_active":"2024-04-25T19:11:34.115519+0000","last_became_peered":"2024-04-25T19:11:34.115519+0000","last_unstale":"2024-04-25T19:11:35.471102+0000","last_undegraded":"2024-04-25T19:11:35.471102+0000","last_fullsized":"2024-04-25T19:11:35.471102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.795134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990163+0000","last_change":"2024-04-25T19:11:29.930630+0000","last_active":"2024-04-25T19:11:36.990163+0000","last_peered":"2024-04-25T19:11:36.990163+0000","last_clean":"2024-04-25T19:11:36.990163+0000","last_became_active":"2024-04-25T19:11:29.930314+0000","last_became_peered":"2024-04-25T19:11:29.930314+0000","last_unstale":"2024-04-25T19:11:36.990163+0000","last_undegraded":"2024-04-25T19:11:36.990163+0000","last_fullsized":"2024-04-25T19:11:36.990163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:44.955421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170921+0000","last_change":"2024-04-25T19:11:28.938687+0000","last_active":"2024-04-25T19:11:37.170921+0000","last_peered":"2024-04-25T19:11:37.170921+0000","last_clean":"2024-04-25T19:11:37.170921+0000","last_became_active":"2024-04-25T19:11:28.938288+0000","last_became_peered":"2024-04-25T19:11:28.938288+0000","last_unstale":"2024-04-25T19:11:37.170921+0000","last_undegraded":"2024-04-25T19:11:37.170921+0000","last_fullsized":"2024-04-25T19:11:37.170921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990666+0000","last_change":"2024-04-25T19:11:33.972056+0000","last_active":"2024-04-25T19:11:36.990666+0000","last_peered":"2024-04-25T19:11:36.990666+0000","last_clean":"2024-04-25T19:11:36.990666+0000","last_became_active":"2024-04-25T19:11:33.971484+0000","last_became_peered":"2024-04-25T19:11:33.971484+0000","last_unstale":"2024-04-25T19:11:36.990666+0000","last_undegraded":"2024-04-25T19:11:36.990666+0000","last_fullsized":"2024-04-25T19:11:36.990666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:52.951275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878644+0000","last_change":"2024-04-25T19:11:29.935281+0000","last_active":"2024-04-25T19:11:35.878644+0000","last_peered":"2024-04-25T19:11:35.878644+0000","last_clean":"2024-04-25T19:11:35.878644+0000","last_became_active":"2024-04-25T19:11:29.934928+0000","last_became_peered":"2024-04-25T19:11:29.934928+0000","last_unstale":"2024-04-25T19:11:35.878644+0000","last_undegraded":"2024-04-25T19:11:35.878644+0000","last_fullsized":"2024-04-25T19:11:35.878644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963243+0000","last_change":"2024-04-25T19:11:28.940436+0000","last_active":"2024-04-25T19:11:34.963243+0000","last_peered":"2024-04-25T19:11:34.963243+0000","last_clean":"2024-04-25T19:11:34.963243+0000","last_became_active":"2024-04-25T19:11:28.940050+0000","last_became_peered":"2024-04-25T19:11:28.940050+0000","last_unstale":"2024-04-25T19:11:34.963243+0000","last_undegraded":"2024-04-25T19:11:34.963243+0000","last_fullsized":"2024-04-25T19:11:34.963243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989678+0000","last_change":"2024-04-25T19:11:34.115813+0000","last_active":"2024-04-25T19:11:36.989678+0000","last_peered":"2024-04-25T19:11:36.989678+0000","last_clean":"2024-04-25T19:11:36.989678+0000","last_became_active":"2024-04-25T19:11:34.115268+0000","last_became_peered":"2024-04-25T19:11:34.115268+0000","last_unstale":"2024-04-25T19:11:36.989678+0000","last_undegraded":"2024-04-25T19:11:36.989678+0000","last_fullsized":"2024-04-25T19:11:36.989678+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:54.194127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171971+0000","last_change":"2024-04-25T19:11:29.945249+0000","last_active":"2024-04-25T19:11:37.171971+0000","last_peered":"2024-04-25T19:11:37.171971+0000","last_clean":"2024-04-25T19:11:37.171971+0000","last_became_active":"2024-04-25T19:11:29.944930+0000","last_became_peered":"2024-04-25T19:11:29.944930+0000","last_unstale":"2024-04-25T19:11:37.171971+0000","last_undegraded":"2024-04-25T19:11:37.171971+0000","last_fullsized":"2024-04-25T19:11:37.171971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470980+0000","last_change":"2024-04-25T19:11:28.937963+0000","last_active":"2024-04-25T19:11:35.470980+0000","last_peered":"2024-04-25T19:11:35.470980+0000","last_clean":"2024-04-25T19:11:35.470980+0000","last_became_active":"2024-04-25T19:11:28.936818+0000","last_became_peered":"2024-04-25T19:11:28.936818+0000","last_unstale":"2024-04-25T19:11:35.470980+0000","last_undegraded":"2024-04-25T19:11:35.470980+0000","last_fullsized":"2024-04-25T19:11:35.470980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170969+0000","last_change":"2024-04-25T19:11:34.113848+0000","last_active":"2024-04-25T19:11:37.170969+0000","last_peered":"2024-04-25T19:11:37.170969+0000","last_clean":"2024-04-25T19:11:37.170969+0000","last_became_active":"2024-04-25T19:11:34.113403+0000","last_became_peered":"2024-04-25T19:11:34.113403+0000","last_unstale":"2024-04-25T19:11:37.170969+0000","last_undegraded":"2024-04-25T19:11:37.170969+0000","last_fullsized":"2024-04-25T19:11:37.170969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:43.702771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470714+0000","last_change":"2024-04-25T19:11:29.930348+0000","last_active":"2024-04-25T19:11:35.470714+0000","last_peered":"2024-04-25T19:11:35.470714+0000","last_clean":"2024-04-25T19:11:35.470714+0000","last_became_active":"2024-04-25T19:11:29.930028+0000","last_became_peered":"2024-04-25T19:11:29.930028+0000","last_unstale":"2024-04-25T19:11:35.470714+0000","last_undegraded":"2024-04-25T19:11:35.470714+0000","last_fullsized":"2024-04-25T19:11:35.470714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170925+0000","last_change":"2024-04-25T19:11:28.939508+0000","last_active":"2024-04-25T19:11:37.170925+0000","last_peered":"2024-04-25T19:11:37.170925+0000","last_clean":"2024-04-25T19:11:37.170925+0000","last_became_active":"2024-04-25T19:11:28.939050+0000","last_became_peered":"2024-04-25T19:11:28.939050+0000","last_unstale":"2024-04-25T19:11:37.170925+0000","last_undegraded":"2024-04-25T19:11:37.170925+0000","last_fullsized":"2024-04-25T19:11:37.170925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:00.743906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878165+0000","last_change":"2024-04-25T19:11:34.110317+0000","last_active":"2024-04-25T19:11:35.878165+0000","last_peered":"2024-04-25T19:11:35.878165+0000","last_clean":"2024-04-25T19:11:35.878165+0000","last_became_active":"2024-04-25T19:11:34.109906+0000","last_became_peered":"2024-04-25T19:11:34.109906+0000","last_unstale":"2024-04-25T19:11:35.878165+0000","last_undegraded":"2024-04-25T19:11:35.878165+0000","last_fullsized":"2024-04-25T19:11:35.878165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:35.687485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962541+0000","last_change":"2024-04-25T19:11:29.952079+0000","last_active":"2024-04-25T19:11:34.962541+0000","last_peered":"2024-04-25T19:11:34.962541+0000","last_clean":"2024-04-25T19:11:34.962541+0000","last_became_active":"2024-04-25T19:11:29.951437+0000","last_became_peered":"2024-04-25T19:11:29.951437+0000","last_unstale":"2024-04-25T19:11:34.962541+0000","last_undegraded":"2024-04-25T19:11:34.962541+0000","last_fullsized":"2024-04-25T19:11:34.962541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470828+0000","last_change":"2024-04-25T19:11:28.933142+0000","last_active":"2024-04-25T19:11:35.470828+0000","last_peered":"2024-04-25T19:11:35.470828+0000","last_clean":"2024-04-25T19:11:35.470828+0000","last_became_active":"2024-04-25T19:11:28.932682+0000","last_became_peered":"2024-04-25T19:11:28.932682+0000","last_unstale":"2024-04-25T19:11:35.470828+0000","last_undegraded":"2024-04-25T19:11:35.470828+0000","last_fullsized":"2024-04-25T19:11:35.470828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:38.236609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470450+0000","last_change":"2024-04-25T19:11:33.971859+0000","last_active":"2024-04-25T19:11:35.470450+0000","last_peered":"2024-04-25T19:11:35.470450+0000","last_clean":"2024-04-25T19:11:35.470450+0000","last_became_active":"2024-04-25T19:11:33.971349+0000","last_became_peered":"2024-04-25T19:11:33.971349+0000","last_unstale":"2024-04-25T19:11:35.470450+0000","last_undegraded":"2024-04-25T19:11:35.470450+0000","last_fullsized":"2024-04-25T19:11:35.470450+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:43.916700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990771+0000","last_change":"2024-04-25T19:11:29.929623+0000","last_active":"2024-04-25T19:11:36.990771+0000","last_peered":"2024-04-25T19:11:36.990771+0000","last_clean":"2024-04-25T19:11:36.990771+0000","last_became_active":"2024-04-25T19:11:29.929306+0000","last_became_peered":"2024-04-25T19:11:29.929306+0000","last_unstale":"2024-04-25T19:11:36.990771+0000","last_undegraded":"2024-04-25T19:11:36.990771+0000","last_fullsized":"2024-04-25T19:11:36.990771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:49.958865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963468+0000","last_change":"2024-04-25T19:11:28.933056+0000","last_active":"2024-04-25T19:11:34.963468+0000","last_peered":"2024-04-25T19:11:34.963468+0000","last_clean":"2024-04-25T19:11:34.963468+0000","last_became_active":"2024-04-25T19:11:28.930233+0000","last_became_peered":"2024-04-25T19:11:28.930233+0000","last_unstale":"2024-04-25T19:11:34.963468+0000","last_undegraded":"2024-04-25T19:11:34.963468+0000","last_fullsized":"2024-04-25T19:11:34.963468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470757+0000","last_change":"2024-04-25T19:11:33.972447+0000","last_active":"2024-04-25T19:11:35.470757+0000","last_peered":"2024-04-25T19:11:35.470757+0000","last_clean":"2024-04-25T19:11:35.470757+0000","last_became_active":"2024-04-25T19:11:33.970645+0000","last_became_peered":"2024-04-25T19:11:33.970645+0000","last_unstale":"2024-04-25T19:11:35.470757+0000","last_undegraded":"2024-04-25T19:11:35.470757+0000","last_fullsized":"2024-04-25T19:11:35.470757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:40.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470256+0000","last_change":"2024-04-25T19:11:29.930967+0000","last_active":"2024-04-25T19:11:35.470256+0000","last_peered":"2024-04-25T19:11:35.470256+0000","last_clean":"2024-04-25T19:11:35.470256+0000","last_became_active":"2024-04-25T19:11:29.930682+0000","last_became_peered":"2024-04-25T19:11:29.930682+0000","last_unstale":"2024-04-25T19:11:35.470256+0000","last_undegraded":"2024-04-25T19:11:35.470256+0000","last_fullsized":"2024-04-25T19:11:35.470256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960304+0000","last_change":"2024-04-25T19:11:28.932205+0000","last_active":"2024-04-25T19:11:34.960304+0000","last_peered":"2024-04-25T19:11:34.960304+0000","last_clean":"2024-04-25T19:11:34.960304+0000","last_became_active":"2024-04-25T19:11:28.931851+0000","last_became_peered":"2024-04-25T19:11:28.931851+0000","last_unstale":"2024-04-25T19:11:34.960304+0000","last_undegraded":"2024-04-25T19:11:34.960304+0000","last_fullsized":"2024-04-25T19:11:34.960304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.471128+0000","last_change":"2024-04-25T19:11:33.971485+0000","last_active":"2024-04-25T19:11:35.471128+0000","last_peered":"2024-04-25T19:11:35.471128+0000","last_clean":"2024-04-25T19:11:35.471128+0000","last_became_active":"2024-04-25T19:11:33.971022+0000","last_became_peered":"2024-04-25T19:11:33.971022+0000","last_unstale":"2024-04-25T19:11:35.471128+0000","last_undegraded":"2024-04-25T19:11:35.471128+0000","last_fullsized":"2024-04-25T19:11:35.471128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:38.690554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878004+0000","last_change":"2024-04-25T19:11:29.936287+0000","last_active":"2024-04-25T19:11:35.878004+0000","last_peered":"2024-04-25T19:11:35.878004+0000","last_clean":"2024-04-25T19:11:35.878004+0000","last_became_active":"2024-04-25T19:11:29.935945+0000","last_became_peered":"2024-04-25T19:11:29.935945+0000","last_unstale":"2024-04-25T19:11:35.878004+0000","last_undegraded":"2024-04-25T19:11:35.878004+0000","last_fullsized":"2024-04-25T19:11:35.878004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:41.846903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170920+0000","last_change":"2024-04-25T19:11:28.940966+0000","last_active":"2024-04-25T19:11:37.170920+0000","last_peered":"2024-04-25T19:11:37.170920+0000","last_clean":"2024-04-25T19:11:37.170920+0000","last_became_active":"2024-04-25T19:11:28.939773+0000","last_became_peered":"2024-04-25T19:11:28.939773+0000","last_unstale":"2024-04-25T19:11:37.170920+0000","last_undegraded":"2024-04-25T19:11:37.170920+0000","last_fullsized":"2024-04-25T19:11:37.170920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878537+0000","last_change":"2024-04-25T19:11:34.113282+0000","last_active":"2024-04-25T19:11:35.878537+0000","last_peered":"2024-04-25T19:11:35.878537+0000","last_clean":"2024-04-25T19:11:35.878537+0000","last_became_active":"2024-04-25T19:11:34.112444+0000","last_became_peered":"2024-04-25T19:11:34.112444+0000","last_unstale":"2024-04-25T19:11:35.878537+0000","last_undegraded":"2024-04-25T19:11:35.878537+0000","last_fullsized":"2024-04-25T19:11:35.878537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:43.702771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170832+0000","last_change":"2024-04-25T19:11:29.949528+0000","last_active":"2024-04-25T19:11:37.170832+0000","last_peered":"2024-04-25T19:11:37.170832+0000","last_clean":"2024-04-25T19:11:37.170832+0000","last_became_active":"2024-04-25T19:11:29.948760+0000","last_became_peered":"2024-04-25T19:11:29.948760+0000","last_unstale":"2024-04-25T19:11:37.170832+0000","last_undegraded":"2024-04-25T19:11:37.170832+0000","last_fullsized":"2024-04-25T19:11:37.170832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963446+0000","last_change":"2024-04-25T19:11:28.928976+0000","last_active":"2024-04-25T19:11:34.963446+0000","last_peered":"2024-04-25T19:11:34.963446+0000","last_clean":"2024-04-25T19:11:34.963446+0000","last_became_active":"2024-04-25T19:11:28.928563+0000","last_became_peered":"2024-04-25T19:11:28.928563+0000","last_unstale":"2024-04-25T19:11:34.963446+0000","last_undegraded":"2024-04-25T19:11:34.963446+0000","last_fullsized":"2024-04-25T19:11:34.963446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":117,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171444+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:11:37.171444+0000","last_peered":"2024-04-25T19:11:37.171444+0000","last_clean":"2024-04-25T19:11:37.171444+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:11:37.171444+0000","last_undegraded":"2024-04-25T19:11:37.171444+0000","last_fullsized":"2024-04-25T19:11:37.171444+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962407+0000","last_change":"2024-04-25T19:11:34.113394+0000","last_active":"2024-04-25T19:11:34.962407+0000","last_peered":"2024-04-25T19:11:34.962407+0000","last_clean":"2024-04-25T19:11:34.962407+0000","last_became_active":"2024-04-25T19:11:34.112950+0000","last_became_peered":"2024-04-25T19:11:34.112950+0000","last_unstale":"2024-04-25T19:11:34.962407+0000","last_undegraded":"2024-04-25T19:11:34.962407+0000","last_fullsized":"2024-04-25T19:11:34.962407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:43.916700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171981+0000","last_change":"2024-04-25T19:11:29.946818+0000","last_active":"2024-04-25T19:11:37.171981+0000","last_peered":"2024-04-25T19:11:37.171981+0000","last_clean":"2024-04-25T19:11:37.171981+0000","last_became_active":"2024-04-25T19:11:29.946517+0000","last_became_peered":"2024-04-25T19:11:29.946517+0000","last_unstale":"2024-04-25T19:11:37.171981+0000","last_undegraded":"2024-04-25T19:11:37.171981+0000","last_fullsized":"2024-04-25T19:11:37.171981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960736+0000","last_change":"2024-04-25T19:11:28.928753+0000","last_active":"2024-04-25T19:11:34.960736+0000","last_peered":"2024-04-25T19:11:34.960736+0000","last_clean":"2024-04-25T19:11:34.960736+0000","last_became_active":"2024-04-25T19:11:28.928388+0000","last_became_peered":"2024-04-25T19:11:28.928388+0000","last_unstale":"2024-04-25T19:11:34.960736+0000","last_undegraded":"2024-04-25T19:11:34.960736+0000","last_fullsized":"2024-04-25T19:11:34.960736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961773+0000","last_change":"2024-04-25T19:11:34.110545+0000","last_active":"2024-04-25T19:11:34.961773+0000","last_peered":"2024-04-25T19:11:34.961773+0000","last_clean":"2024-04-25T19:11:34.961773+0000","last_became_active":"2024-04-25T19:11:34.110024+0000","last_became_peered":"2024-04-25T19:11:34.110024+0000","last_unstale":"2024-04-25T19:11:34.961773+0000","last_undegraded":"2024-04-25T19:11:34.961773+0000","last_fullsized":"2024-04-25T19:11:34.961773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:46.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962527+0000","last_change":"2024-04-25T19:11:29.950030+0000","last_active":"2024-04-25T19:11:34.962527+0000","last_peered":"2024-04-25T19:11:34.962527+0000","last_clean":"2024-04-25T19:11:34.962527+0000","last_became_active":"2024-04-25T19:11:29.949424+0000","last_became_peered":"2024-04-25T19:11:29.949424+0000","last_unstale":"2024-04-25T19:11:34.962527+0000","last_undegraded":"2024-04-25T19:11:34.962527+0000","last_fullsized":"2024-04-25T19:11:34.962527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:48.917939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470828+0000","last_change":"2024-04-25T19:11:28.934208+0000","last_active":"2024-04-25T19:11:35.470828+0000","last_peered":"2024-04-25T19:11:35.470828+0000","last_clean":"2024-04-25T19:11:35.470828+0000","last_became_active":"2024-04-25T19:11:28.933866+0000","last_became_peered":"2024-04-25T19:11:28.933866+0000","last_unstale":"2024-04-25T19:11:35.470828+0000","last_undegraded":"2024-04-25T19:11:35.470828+0000","last_fullsized":"2024-04-25T19:11:35.470828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:30.900062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878372+0000","last_change":"2024-04-25T19:11:33.980318+0000","last_active":"2024-04-25T19:11:35.878372+0000","last_peered":"2024-04-25T19:11:35.878372+0000","last_clean":"2024-04-25T19:11:35.878372+0000","last_became_active":"2024-04-25T19:11:33.979818+0000","last_became_peered":"2024-04-25T19:11:33.979818+0000","last_unstale":"2024-04-25T19:11:35.878372+0000","last_undegraded":"2024-04-25T19:11:35.878372+0000","last_fullsized":"2024-04-25T19:11:35.878372+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:38.690554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172468+0000","last_change":"2024-04-25T19:11:29.940479+0000","last_active":"2024-04-25T19:11:37.172468+0000","last_peered":"2024-04-25T19:11:37.172468+0000","last_clean":"2024-04-25T19:11:37.172468+0000","last_became_active":"2024-04-25T19:11:29.940140+0000","last_became_peered":"2024-04-25T19:11:29.940140+0000","last_unstale":"2024-04-25T19:11:37.172468+0000","last_undegraded":"2024-04-25T19:11:37.172468+0000","last_fullsized":"2024-04-25T19:11:37.172468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:56.761797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960699+0000","last_change":"2024-04-25T19:11:28.933352+0000","last_active":"2024-04-25T19:11:34.960699+0000","last_peered":"2024-04-25T19:11:34.960699+0000","last_clean":"2024-04-25T19:11:34.960699+0000","last_became_active":"2024-04-25T19:11:28.933016+0000","last_became_peered":"2024-04-25T19:11:28.933016+0000","last_unstale":"2024-04-25T19:11:34.960699+0000","last_undegraded":"2024-04-25T19:11:34.960699+0000","last_fullsized":"2024-04-25T19:11:34.960699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961463+0000","last_change":"2024-04-25T19:11:34.110567+0000","last_active":"2024-04-25T19:11:34.961463+0000","last_peered":"2024-04-25T19:11:34.961463+0000","last_clean":"2024-04-25T19:11:34.961463+0000","last_became_active":"2024-04-25T19:11:34.110074+0000","last_became_peered":"2024-04-25T19:11:34.110074+0000","last_unstale":"2024-04-25T19:11:34.961463+0000","last_undegraded":"2024-04-25T19:11:34.961463+0000","last_fullsized":"2024-04-25T19:11:34.961463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:54.738252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172199+0000","last_change":"2024-04-25T19:11:29.939683+0000","last_active":"2024-04-25T19:11:37.172199+0000","last_peered":"2024-04-25T19:11:37.172199+0000","last_clean":"2024-04-25T19:11:37.172199+0000","last_became_active":"2024-04-25T19:11:29.939372+0000","last_became_peered":"2024-04-25T19:11:29.939372+0000","last_unstale":"2024-04-25T19:11:37.172199+0000","last_undegraded":"2024-04-25T19:11:37.172199+0000","last_fullsized":"2024-04-25T19:11:37.172199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:51.175553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989410+0000","last_change":"2024-04-25T19:11:28.928627+0000","last_active":"2024-04-25T19:11:36.989410+0000","last_peered":"2024-04-25T19:11:36.989410+0000","last_clean":"2024-04-25T19:11:36.989410+0000","last_became_active":"2024-04-25T19:11:28.928016+0000","last_became_peered":"2024-04-25T19:11:28.928016+0000","last_unstale":"2024-04-25T19:11:36.989410+0000","last_undegraded":"2024-04-25T19:11:36.989410+0000","last_fullsized":"2024-04-25T19:11:36.989410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:16.833337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989757+0000","last_change":"2024-04-25T19:11:33.971762+0000","last_active":"2024-04-25T19:11:36.989757+0000","last_peered":"2024-04-25T19:11:36.989757+0000","last_clean":"2024-04-25T19:11:36.989757+0000","last_became_active":"2024-04-25T19:11:33.971317+0000","last_became_peered":"2024-04-25T19:11:33.971317+0000","last_unstale":"2024-04-25T19:11:36.989757+0000","last_undegraded":"2024-04-25T19:11:36.989757+0000","last_fullsized":"2024-04-25T19:11:36.989757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:23.748216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171561+0000","last_change":"2024-04-25T19:11:29.938906+0000","last_active":"2024-04-25T19:11:37.171561+0000","last_peered":"2024-04-25T19:11:37.171561+0000","last_clean":"2024-04-25T19:11:37.171561+0000","last_became_active":"2024-04-25T19:11:29.938575+0000","last_became_peered":"2024-04-25T19:11:29.938575+0000","last_unstale":"2024-04-25T19:11:37.171561+0000","last_undegraded":"2024-04-25T19:11:37.171561+0000","last_fullsized":"2024-04-25T19:11:37.171561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960132+0000","last_change":"2024-04-25T19:11:28.931773+0000","last_active":"2024-04-25T19:11:34.960132+0000","last_peered":"2024-04-25T19:11:34.960132+0000","last_clean":"2024-04-25T19:11:34.960132+0000","last_became_active":"2024-04-25T19:11:28.931409+0000","last_became_peered":"2024-04-25T19:11:28.931409+0000","last_unstale":"2024-04-25T19:11:34.960132+0000","last_undegraded":"2024-04-25T19:11:34.960132+0000","last_fullsized":"2024-04-25T19:11:34.960132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:27:30.904833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878361+0000","last_change":"2024-04-25T19:11:33.975482+0000","last_active":"2024-04-25T19:11:35.878361+0000","last_peered":"2024-04-25T19:11:35.878361+0000","last_clean":"2024-04-25T19:11:35.878361+0000","last_became_active":"2024-04-25T19:11:33.975042+0000","last_became_peered":"2024-04-25T19:11:33.975042+0000","last_unstale":"2024-04-25T19:11:35.878361+0000","last_undegraded":"2024-04-25T19:11:35.878361+0000","last_fullsized":"2024-04-25T19:11:35.878361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.795134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961644+0000","last_change":"2024-04-25T19:11:29.943783+0000","last_active":"2024-04-25T19:11:34.961644+0000","last_peered":"2024-04-25T19:11:34.961644+0000","last_clean":"2024-04-25T19:11:34.961644+0000","last_became_active":"2024-04-25T19:11:29.942647+0000","last_became_peered":"2024-04-25T19:11:29.942647+0000","last_unstale":"2024-04-25T19:11:34.961644+0000","last_undegraded":"2024-04-25T19:11:34.961644+0000","last_fullsized":"2024-04-25T19:11:34.961644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961794+0000","last_change":"2024-04-25T19:11:28.934715+0000","last_active":"2024-04-25T19:11:34.961794+0000","last_peered":"2024-04-25T19:11:34.961794+0000","last_clean":"2024-04-25T19:11:34.961794+0000","last_became_active":"2024-04-25T19:11:28.934331+0000","last_became_peered":"2024-04-25T19:11:28.934331+0000","last_unstale":"2024-04-25T19:11:34.961794+0000","last_undegraded":"2024-04-25T19:11:34.961794+0000","last_fullsized":"2024-04-25T19:11:34.961794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171897+0000","last_change":"2024-04-25T19:11:34.115885+0000","last_active":"2024-04-25T19:11:37.171897+0000","last_peered":"2024-04-25T19:11:37.171897+0000","last_clean":"2024-04-25T19:11:37.171897+0000","last_became_active":"2024-04-25T19:11:34.115489+0000","last_became_peered":"2024-04-25T19:11:34.115489+0000","last_unstale":"2024-04-25T19:11:37.171897+0000","last_undegraded":"2024-04-25T19:11:37.171897+0000","last_fullsized":"2024-04-25T19:11:37.171897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:15.088779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171931+0000","last_change":"2024-04-25T19:11:29.937605+0000","last_active":"2024-04-25T19:11:37.171931+0000","last_peered":"2024-04-25T19:11:37.171931+0000","last_clean":"2024-04-25T19:11:37.171931+0000","last_became_active":"2024-04-25T19:11:29.937045+0000","last_became_peered":"2024-04-25T19:11:29.937045+0000","last_unstale":"2024-04-25T19:11:37.171931+0000","last_undegraded":"2024-04-25T19:11:37.171931+0000","last_fullsized":"2024-04-25T19:11:37.171931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:39.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963413+0000","last_change":"2024-04-25T19:11:28.937550+0000","last_active":"2024-04-25T19:11:34.963413+0000","last_peered":"2024-04-25T19:11:34.963413+0000","last_clean":"2024-04-25T19:11:34.963413+0000","last_became_active":"2024-04-25T19:11:28.937224+0000","last_became_peered":"2024-04-25T19:11:28.937224+0000","last_unstale":"2024-04-25T19:11:34.963413+0000","last_undegraded":"2024-04-25T19:11:34.963413+0000","last_fullsized":"2024-04-25T19:11:34.963413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171513+0000","last_change":"2024-04-25T19:11:34.112538+0000","last_active":"2024-04-25T19:11:37.171513+0000","last_peered":"2024-04-25T19:11:37.171513+0000","last_clean":"2024-04-25T19:11:37.171513+0000","last_became_active":"2024-04-25T19:11:34.112105+0000","last_became_peered":"2024-04-25T19:11:34.112105+0000","last_unstale":"2024-04-25T19:11:37.171513+0000","last_undegraded":"2024-04-25T19:11:37.171513+0000","last_fullsized":"2024-04-25T19:11:37.171513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:04.753913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878413+0000","last_change":"2024-04-25T19:11:29.940791+0000","last_active":"2024-04-25T19:11:35.878413+0000","last_peered":"2024-04-25T19:11:35.878413+0000","last_clean":"2024-04-25T19:11:35.878413+0000","last_became_active":"2024-04-25T19:11:29.940494+0000","last_became_peered":"2024-04-25T19:11:29.940494+0000","last_unstale":"2024-04-25T19:11:35.878413+0000","last_undegraded":"2024-04-25T19:11:35.878413+0000","last_fullsized":"2024-04-25T19:11:35.878413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878662+0000","last_change":"2024-04-25T19:11:28.931517+0000","last_active":"2024-04-25T19:11:35.878662+0000","last_peered":"2024-04-25T19:11:35.878662+0000","last_clean":"2024-04-25T19:11:35.878662+0000","last_became_active":"2024-04-25T19:11:28.930964+0000","last_became_peered":"2024-04-25T19:11:28.930964+0000","last_unstale":"2024-04-25T19:11:35.878662+0000","last_undegraded":"2024-04-25T19:11:35.878662+0000","last_fullsized":"2024-04-25T19:11:35.878662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:14.977696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962359+0000","last_change":"2024-04-25T19:11:34.113386+0000","last_active":"2024-04-25T19:11:34.962359+0000","last_peered":"2024-04-25T19:11:34.962359+0000","last_clean":"2024-04-25T19:11:34.962359+0000","last_became_active":"2024-04-25T19:11:34.112820+0000","last_became_peered":"2024-04-25T19:11:34.112820+0000","last_unstale":"2024-04-25T19:11:34.962359+0000","last_undegraded":"2024-04-25T19:11:34.962359+0000","last_fullsized":"2024-04-25T19:11:34.962359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:04.753913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171892+0000","last_change":"2024-04-25T19:11:29.933966+0000","last_active":"2024-04-25T19:11:37.171892+0000","last_peered":"2024-04-25T19:11:37.171892+0000","last_clean":"2024-04-25T19:11:37.171892+0000","last_became_active":"2024-04-25T19:11:29.933549+0000","last_became_peered":"2024-04-25T19:11:29.933549+0000","last_unstale":"2024-04-25T19:11:37.171892+0000","last_undegraded":"2024-04-25T19:11:37.171892+0000","last_fullsized":"2024-04-25T19:11:37.171892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:39.517432+0000","last_change":"2024-04-25T19:11:28.936547+0000","last_active":"2024-04-25T19:11:39.517432+0000","last_peered":"2024-04-25T19:11:39.517432+0000","last_clean":"2024-04-25T19:11:39.517432+0000","last_became_active":"2024-04-25T19:11:28.935816+0000","last_became_peered":"2024-04-25T19:11:28.935816+0000","last_unstale":"2024-04-25T19:11:39.517432+0000","last_undegraded":"2024-04-25T19:11:39.517432+0000","last_fullsized":"2024-04-25T19:11:39.517432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470655+0000","last_change":"2024-04-25T19:11:33.972776+0000","last_active":"2024-04-25T19:11:35.470655+0000","last_peered":"2024-04-25T19:11:35.470655+0000","last_clean":"2024-04-25T19:11:35.470655+0000","last_became_active":"2024-04-25T19:11:33.971757+0000","last_became_peered":"2024-04-25T19:11:33.971757+0000","last_unstale":"2024-04-25T19:11:35.470655+0000","last_undegraded":"2024-04-25T19:11:35.470655+0000","last_fullsized":"2024-04-25T19:11:35.470655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:35.687485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990580+0000","last_change":"2024-04-25T19:11:29.925691+0000","last_active":"2024-04-25T19:11:36.990580+0000","last_peered":"2024-04-25T19:11:36.990580+0000","last_clean":"2024-04-25T19:11:36.990580+0000","last_became_active":"2024-04-25T19:11:29.924990+0000","last_became_peered":"2024-04-25T19:11:29.924990+0000","last_unstale":"2024-04-25T19:11:36.990580+0000","last_undegraded":"2024-04-25T19:11:36.990580+0000","last_fullsized":"2024-04-25T19:11:36.990580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:39.242588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470697+0000","last_change":"2024-04-25T19:11:28.937077+0000","last_active":"2024-04-25T19:11:35.470697+0000","last_peered":"2024-04-25T19:11:35.470697+0000","last_clean":"2024-04-25T19:11:35.470697+0000","last_became_active":"2024-04-25T19:11:28.936181+0000","last_became_peered":"2024-04-25T19:11:28.936181+0000","last_unstale":"2024-04-25T19:11:35.470697+0000","last_undegraded":"2024-04-25T19:11:35.470697+0000","last_fullsized":"2024-04-25T19:11:35.470697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990233+0000","last_change":"2024-04-25T19:11:33.972375+0000","last_active":"2024-04-25T19:11:36.990233+0000","last_peered":"2024-04-25T19:11:36.990233+0000","last_clean":"2024-04-25T19:11:36.990233+0000","last_became_active":"2024-04-25T19:11:33.971792+0000","last_became_peered":"2024-04-25T19:11:33.971792+0000","last_unstale":"2024-04-25T19:11:36.990233+0000","last_undegraded":"2024-04-25T19:11:36.990233+0000","last_fullsized":"2024-04-25T19:11:36.990233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:13.742925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878422+0000","last_change":"2024-04-25T19:11:29.943424+0000","last_active":"2024-04-25T19:11:35.878422+0000","last_peered":"2024-04-25T19:11:35.878422+0000","last_clean":"2024-04-25T19:11:35.878422+0000","last_became_active":"2024-04-25T19:11:29.943096+0000","last_became_peered":"2024-04-25T19:11:29.943096+0000","last_unstale":"2024-04-25T19:11:35.878422+0000","last_undegraded":"2024-04-25T19:11:35.878422+0000","last_fullsized":"2024-04-25T19:11:35.878422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962894+0000","last_change":"2024-04-25T19:11:28.934405+0000","last_active":"2024-04-25T19:11:34.962894+0000","last_peered":"2024-04-25T19:11:34.962894+0000","last_clean":"2024-04-25T19:11:34.962894+0000","last_became_active":"2024-04-25T19:11:28.933743+0000","last_became_peered":"2024-04-25T19:11:28.933743+0000","last_unstale":"2024-04-25T19:11:34.962894+0000","last_undegraded":"2024-04-25T19:11:34.962894+0000","last_fullsized":"2024-04-25T19:11:34.962894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171201+0000","last_change":"2024-04-25T19:11:33.969689+0000","last_active":"2024-04-25T19:11:37.171201+0000","last_peered":"2024-04-25T19:11:37.171201+0000","last_clean":"2024-04-25T19:11:37.171201+0000","last_became_active":"2024-04-25T19:11:33.969257+0000","last_became_peered":"2024-04-25T19:11:33.969257+0000","last_unstale":"2024-04-25T19:11:37.171201+0000","last_undegraded":"2024-04-25T19:11:37.171201+0000","last_fullsized":"2024-04-25T19:11:37.171201+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:57.736550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962757+0000","last_change":"2024-04-25T19:11:29.949944+0000","last_active":"2024-04-25T19:11:34.962757+0000","last_peered":"2024-04-25T19:11:34.962757+0000","last_clean":"2024-04-25T19:11:34.962757+0000","last_became_active":"2024-04-25T19:11:29.949521+0000","last_became_peered":"2024-04-25T19:11:29.949521+0000","last_unstale":"2024-04-25T19:11:34.962757+0000","last_undegraded":"2024-04-25T19:11:34.962757+0000","last_fullsized":"2024-04-25T19:11:34.962757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963008+0000","last_change":"2024-04-25T19:11:28.942291+0000","last_active":"2024-04-25T19:11:34.963008+0000","last_peered":"2024-04-25T19:11:34.963008+0000","last_clean":"2024-04-25T19:11:34.963008+0000","last_became_active":"2024-04-25T19:11:28.941951+0000","last_became_peered":"2024-04-25T19:11:28.941951+0000","last_unstale":"2024-04-25T19:11:34.963008+0000","last_undegraded":"2024-04-25T19:11:34.963008+0000","last_fullsized":"2024-04-25T19:11:34.963008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171184+0000","last_change":"2024-04-25T19:11:34.114829+0000","last_active":"2024-04-25T19:11:37.171184+0000","last_peered":"2024-04-25T19:11:37.171184+0000","last_clean":"2024-04-25T19:11:37.171184+0000","last_became_active":"2024-04-25T19:11:34.114452+0000","last_became_peered":"2024-04-25T19:11:34.114452+0000","last_unstale":"2024-04-25T19:11:37.171184+0000","last_undegraded":"2024-04-25T19:11:37.171184+0000","last_fullsized":"2024-04-25T19:11:37.171184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:54.738252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878339+0000","last_change":"2024-04-25T19:11:34.114794+0000","last_active":"2024-04-25T19:11:35.878339+0000","last_peered":"2024-04-25T19:11:35.878339+0000","last_clean":"2024-04-25T19:11:35.878339+0000","last_became_active":"2024-04-25T19:11:34.114376+0000","last_became_peered":"2024-04-25T19:11:34.114376+0000","last_unstale":"2024-04-25T19:11:35.878339+0000","last_undegraded":"2024-04-25T19:11:35.878339+0000","last_fullsized":"2024-04-25T19:11:35.878339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:40.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470153+0000","last_change":"2024-04-25T19:11:29.935850+0000","last_active":"2024-04-25T19:11:35.470153+0000","last_peered":"2024-04-25T19:11:35.470153+0000","last_clean":"2024-04-25T19:11:35.470153+0000","last_became_active":"2024-04-25T19:11:29.935546+0000","last_became_peered":"2024-04-25T19:11:29.935546+0000","last_unstale":"2024-04-25T19:11:35.470153+0000","last_undegraded":"2024-04-25T19:11:35.470153+0000","last_fullsized":"2024-04-25T19:11:35.470153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961554+0000","last_change":"2024-04-25T19:11:28.928204+0000","last_active":"2024-04-25T19:11:34.961554+0000","last_peered":"2024-04-25T19:11:34.961554+0000","last_clean":"2024-04-25T19:11:34.961554+0000","last_became_active":"2024-04-25T19:11:28.927812+0000","last_became_peered":"2024-04-25T19:11:28.927812+0000","last_unstale":"2024-04-25T19:11:34.961554+0000","last_undegraded":"2024-04-25T19:11:34.961554+0000","last_fullsized":"2024-04-25T19:11:34.961554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961976+0000","last_change":"2024-04-25T19:11:33.971937+0000","last_active":"2024-04-25T19:11:34.961976+0000","last_peered":"2024-04-25T19:11:34.961976+0000","last_clean":"2024-04-25T19:11:34.961976+0000","last_became_active":"2024-04-25T19:11:33.971297+0000","last_became_peered":"2024-04-25T19:11:33.971297+0000","last_unstale":"2024-04-25T19:11:34.961976+0000","last_undegraded":"2024-04-25T19:11:34.961976+0000","last_fullsized":"2024-04-25T19:11:34.961976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:52.951275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171272+0000","last_change":"2024-04-25T19:11:29.936690+0000","last_active":"2024-04-25T19:11:37.171272+0000","last_peered":"2024-04-25T19:11:37.171272+0000","last_clean":"2024-04-25T19:11:37.171272+0000","last_became_active":"2024-04-25T19:11:29.934968+0000","last_became_peered":"2024-04-25T19:11:29.934968+0000","last_unstale":"2024-04-25T19:11:37.171272+0000","last_undegraded":"2024-04-25T19:11:37.171272+0000","last_fullsized":"2024-04-25T19:11:37.171272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170564+0000","last_change":"2024-04-25T19:11:28.936664+0000","last_active":"2024-04-25T19:11:37.170564+0000","last_peered":"2024-04-25T19:11:37.170564+0000","last_clean":"2024-04-25T19:11:37.170564+0000","last_became_active":"2024-04-25T19:11:28.936236+0000","last_became_peered":"2024-04-25T19:11:28.936236+0000","last_unstale":"2024-04-25T19:11:37.170564+0000","last_undegraded":"2024-04-25T19:11:37.170564+0000","last_fullsized":"2024-04-25T19:11:37.170564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962069+0000","last_change":"2024-04-25T19:11:28.934644+0000","last_active":"2024-04-25T19:11:34.962069+0000","last_peered":"2024-04-25T19:11:34.962069+0000","last_clean":"2024-04-25T19:11:34.962069+0000","last_became_active":"2024-04-25T19:11:28.934282+0000","last_became_peered":"2024-04-25T19:11:28.934282+0000","last_unstale":"2024-04-25T19:11:34.962069+0000","last_undegraded":"2024-04-25T19:11:34.962069+0000","last_fullsized":"2024-04-25T19:11:34.962069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878411+0000","last_change":"2024-04-25T19:11:29.934419+0000","last_active":"2024-04-25T19:11:35.878411+0000","last_peered":"2024-04-25T19:11:35.878411+0000","last_clean":"2024-04-25T19:11:35.878411+0000","last_became_active":"2024-04-25T19:11:29.934119+0000","last_became_peered":"2024-04-25T19:11:29.934119+0000","last_unstale":"2024-04-25T19:11:35.878411+0000","last_undegraded":"2024-04-25T19:11:35.878411+0000","last_fullsized":"2024-04-25T19:11:35.878411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960537+0000","last_change":"2024-04-25T19:11:33.969976+0000","last_active":"2024-04-25T19:11:34.960537+0000","last_peered":"2024-04-25T19:11:34.960537+0000","last_clean":"2024-04-25T19:11:34.960537+0000","last_became_active":"2024-04-25T19:11:33.969493+0000","last_became_peered":"2024-04-25T19:11:33.969493+0000","last_unstale":"2024-04-25T19:11:34.960537+0000","last_undegraded":"2024-04-25T19:11:34.960537+0000","last_fullsized":"2024-04-25T19:11:34.960537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:45.880240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171502+0000","last_change":"2024-04-25T19:11:28.933487+0000","last_active":"2024-04-25T19:11:37.171502+0000","last_peered":"2024-04-25T19:11:37.171502+0000","last_clean":"2024-04-25T19:11:37.171502+0000","last_became_active":"2024-04-25T19:11:28.933112+0000","last_became_peered":"2024-04-25T19:11:28.933112+0000","last_unstale":"2024-04-25T19:11:37.171502+0000","last_undegraded":"2024-04-25T19:11:37.171502+0000","last_fullsized":"2024-04-25T19:11:37.171502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:44.780178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990049+0000","last_change":"2024-04-25T19:11:29.927210+0000","last_active":"2024-04-25T19:11:36.990049+0000","last_peered":"2024-04-25T19:11:36.990049+0000","last_clean":"2024-04-25T19:11:36.990049+0000","last_became_active":"2024-04-25T19:11:29.926143+0000","last_became_peered":"2024-04-25T19:11:29.926143+0000","last_unstale":"2024-04-25T19:11:36.990049+0000","last_undegraded":"2024-04-25T19:11:36.990049+0000","last_fullsized":"2024-04-25T19:11:36.990049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:31.906041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171522+0000","last_change":"2024-04-25T19:11:33.974927+0000","last_active":"2024-04-25T19:11:37.171522+0000","last_peered":"2024-04-25T19:11:37.171522+0000","last_clean":"2024-04-25T19:11:37.171522+0000","last_became_active":"2024-04-25T19:11:33.974526+0000","last_became_peered":"2024-04-25T19:11:33.974526+0000","last_unstale":"2024-04-25T19:11:37.171522+0000","last_undegraded":"2024-04-25T19:11:37.171522+0000","last_fullsized":"2024-04-25T19:11:37.171522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:32.367757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961363+0000","last_change":"2024-04-25T19:11:29.943955+0000","last_active":"2024-04-25T19:11:34.961363+0000","last_peered":"2024-04-25T19:11:34.961363+0000","last_clean":"2024-04-25T19:11:34.961363+0000","last_became_active":"2024-04-25T19:11:29.943620+0000","last_became_peered":"2024-04-25T19:11:29.943620+0000","last_unstale":"2024-04-25T19:11:34.961363+0000","last_undegraded":"2024-04-25T19:11:34.961363+0000","last_fullsized":"2024-04-25T19:11:34.961363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960143+0000","last_change":"2024-04-25T19:11:29.944224+0000","last_active":"2024-04-25T19:11:34.960143+0000","last_peered":"2024-04-25T19:11:34.960143+0000","last_clean":"2024-04-25T19:11:34.960143+0000","last_became_active":"2024-04-25T19:11:29.943851+0000","last_became_peered":"2024-04-25T19:11:29.943851+0000","last_unstale":"2024-04-25T19:11:34.960143+0000","last_undegraded":"2024-04-25T19:11:34.960143+0000","last_fullsized":"2024-04-25T19:11:34.960143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:31.906041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963271+0000","last_change":"2024-04-25T19:11:29.948559+0000","last_active":"2024-04-25T19:11:34.963271+0000","last_peered":"2024-04-25T19:11:34.963271+0000","last_clean":"2024-04-25T19:11:34.963271+0000","last_became_active":"2024-04-25T19:11:29.948261+0000","last_became_peered":"2024-04-25T19:11:29.948261+0000","last_unstale":"2024-04-25T19:11:34.963271+0000","last_undegraded":"2024-04-25T19:11:34.963271+0000","last_fullsized":"2024-04-25T19:11:34.963271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878297+0000","last_change":"2024-04-25T19:11:29.941924+0000","last_active":"2024-04-25T19:11:35.878297+0000","last_peered":"2024-04-25T19:11:35.878297+0000","last_clean":"2024-04-25T19:11:35.878297+0000","last_became_active":"2024-04-25T19:11:29.939317+0000","last_became_peered":"2024-04-25T19:11:29.939317+0000","last_unstale":"2024-04-25T19:11:35.878297+0000","last_undegraded":"2024-04-25T19:11:35.878297+0000","last_fullsized":"2024-04-25T19:11:35.878297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171676+0000","last_change":"2024-04-25T19:11:29.937716+0000","last_active":"2024-04-25T19:11:37.171676+0000","last_peered":"2024-04-25T19:11:37.171676+0000","last_clean":"2024-04-25T19:11:37.171676+0000","last_became_active":"2024-04-25T19:11:29.936902+0000","last_became_peered":"2024-04-25T19:11:29.936902+0000","last_unstale":"2024-04-25T19:11:37.171676+0000","last_undegraded":"2024-04-25T19:11:37.171676+0000","last_fullsized":"2024-04-25T19:11:37.171676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470475+0000","last_change":"2024-04-25T19:11:29.933948+0000","last_active":"2024-04-25T19:11:35.470475+0000","last_peered":"2024-04-25T19:11:35.470475+0000","last_clean":"2024-04-25T19:11:35.470475+0000","last_became_active":"2024-04-25T19:11:29.933609+0000","last_became_peered":"2024-04-25T19:11:29.933609+0000","last_unstale":"2024-04-25T19:11:35.470475+0000","last_undegraded":"2024-04-25T19:11:35.470475+0000","last_fullsized":"2024-04-25T19:11:35.470475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963143+0000","last_change":"2024-04-25T19:11:29.949696+0000","last_active":"2024-04-25T19:11:34.963143+0000","last_peered":"2024-04-25T19:11:34.963143+0000","last_clean":"2024-04-25T19:11:34.963143+0000","last_became_active":"2024-04-25T19:11:29.949244+0000","last_became_peered":"2024-04-25T19:11:29.949244+0000","last_unstale":"2024-04-25T19:11:34.963143+0000","last_undegraded":"2024-04-25T19:11:34.963143+0000","last_fullsized":"2024-04-25T19:11:34.963143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171872+0000","last_change":"2024-04-25T19:11:29.936507+0000","last_active":"2024-04-25T19:11:37.171872+0000","last_peered":"2024-04-25T19:11:37.171872+0000","last_clean":"2024-04-25T19:11:37.171872+0000","last_became_active":"2024-04-25T19:11:29.936194+0000","last_became_peered":"2024-04-25T19:11:29.936194+0000","last_unstale":"2024-04-25T19:11:37.171872+0000","last_undegraded":"2024-04-25T19:11:37.171872+0000","last_fullsized":"2024-04-25T19:11:37.171872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961467+0000","last_change":"2024-04-25T19:11:29.949296+0000","last_active":"2024-04-25T19:11:34.961467+0000","last_peered":"2024-04-25T19:11:34.961467+0000","last_clean":"2024-04-25T19:11:34.961467+0000","last_became_active":"2024-04-25T19:11:29.948939+0000","last_became_peered":"2024-04-25T19:11:29.948939+0000","last_unstale":"2024-04-25T19:11:34.961467+0000","last_undegraded":"2024-04-25T19:11:34.961467+0000","last_fullsized":"2024-04-25T19:11:34.961467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:39.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171201+0000","last_change":"2024-04-25T19:11:29.937245+0000","last_active":"2024-04-25T19:11:37.171201+0000","last_peered":"2024-04-25T19:11:37.171201+0000","last_clean":"2024-04-25T19:11:37.171201+0000","last_became_active":"2024-04-25T19:11:29.936924+0000","last_became_peered":"2024-04-25T19:11:29.936924+0000","last_unstale":"2024-04-25T19:11:37.171201+0000","last_undegraded":"2024-04-25T19:11:37.171201+0000","last_fullsized":"2024-04-25T19:11:37.171201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963148+0000","last_change":"2024-04-25T19:11:29.939024+0000","last_active":"2024-04-25T19:11:34.963148+0000","last_peered":"2024-04-25T19:11:34.963148+0000","last_clean":"2024-04-25T19:11:34.963148+0000","last_became_active":"2024-04-25T19:11:29.938702+0000","last_became_peered":"2024-04-25T19:11:29.938702+0000","last_unstale":"2024-04-25T19:11:34.963148+0000","last_undegraded":"2024-04-25T19:11:34.963148+0000","last_fullsized":"2024-04-25T19:11:34.963148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878155+0000","last_change":"2024-04-25T19:11:29.942883+0000","last_active":"2024-04-25T19:11:35.878155+0000","last_peered":"2024-04-25T19:11:35.878155+0000","last_clean":"2024-04-25T19:11:35.878155+0000","last_became_active":"2024-04-25T19:11:29.942371+0000","last_became_peered":"2024-04-25T19:11:29.942371+0000","last_unstale":"2024-04-25T19:11:35.878155+0000","last_undegraded":"2024-04-25T19:11:35.878155+0000","last_fullsized":"2024-04-25T19:11:35.878155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962242+0000","last_change":"2024-04-25T19:11:29.940287+0000","last_active":"2024-04-25T19:11:34.962242+0000","last_peered":"2024-04-25T19:11:34.962242+0000","last_clean":"2024-04-25T19:11:34.962242+0000","last_became_active":"2024-04-25T19:11:29.939973+0000","last_became_peered":"2024-04-25T19:11:29.939973+0000","last_unstale":"2024-04-25T19:11:34.962242+0000","last_undegraded":"2024-04-25T19:11:34.962242+0000","last_fullsized":"2024-04-25T19:11:34.962242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172189+0000","last_change":"2024-04-25T19:11:29.932331+0000","last_active":"2024-04-25T19:11:37.172189+0000","last_peered":"2024-04-25T19:11:37.172189+0000","last_clean":"2024-04-25T19:11:37.172189+0000","last_became_active":"2024-04-25T19:11:29.932030+0000","last_became_peered":"2024-04-25T19:11:29.932030+0000","last_unstale":"2024-04-25T19:11:37.172189+0000","last_undegraded":"2024-04-25T19:11:37.172189+0000","last_fullsized":"2024-04-25T19:11:37.172189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:48.917939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960501+0000","last_change":"2024-04-25T19:11:29.930099+0000","last_active":"2024-04-25T19:11:34.960501+0000","last_peered":"2024-04-25T19:11:34.960501+0000","last_clean":"2024-04-25T19:11:34.960501+0000","last_became_active":"2024-04-25T19:11:29.929729+0000","last_became_peered":"2024-04-25T19:11:29.929729+0000","last_unstale":"2024-04-25T19:11:34.960501+0000","last_undegraded":"2024-04-25T19:11:34.960501+0000","last_fullsized":"2024-04-25T19:11:34.960501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:39.242588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963790+0000","last_change":"2024-04-25T19:11:29.940716+0000","last_active":"2024-04-25T19:11:34.963790+0000","last_peered":"2024-04-25T19:11:34.963790+0000","last_clean":"2024-04-25T19:11:34.963790+0000","last_became_active":"2024-04-25T19:11:29.940412+0000","last_became_peered":"2024-04-25T19:11:29.940412+0000","last_unstale":"2024-04-25T19:11:34.963790+0000","last_undegraded":"2024-04-25T19:11:34.963790+0000","last_fullsized":"2024-04-25T19:11:34.963790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:35.993870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961354+0000","last_change":"2024-04-25T19:11:29.945896+0000","last_active":"2024-04-25T19:11:34.961354+0000","last_peered":"2024-04-25T19:11:34.961354+0000","last_clean":"2024-04-25T19:11:34.961354+0000","last_became_active":"2024-04-25T19:11:29.945590+0000","last_became_peered":"2024-04-25T19:11:29.945590+0000","last_unstale":"2024-04-25T19:11:34.961354+0000","last_undegraded":"2024-04-25T19:11:34.961354+0000","last_fullsized":"2024-04-25T19:11:34.961354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:42.722556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470247+0000","last_change":"2024-04-25T19:11:29.939162+0000","last_active":"2024-04-25T19:11:35.470247+0000","last_peered":"2024-04-25T19:11:35.470247+0000","last_clean":"2024-04-25T19:11:35.470247+0000","last_became_active":"2024-04-25T19:11:29.938821+0000","last_became_peered":"2024-04-25T19:11:29.938821+0000","last_unstale":"2024-04-25T19:11:35.470247+0000","last_undegraded":"2024-04-25T19:11:35.470247+0000","last_fullsized":"2024-04-25T19:11:35.470247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878300+0000","last_change":"2024-04-25T19:11:29.938034+0000","last_active":"2024-04-25T19:11:35.878300+0000","last_peered":"2024-04-25T19:11:35.878300+0000","last_clean":"2024-04-25T19:11:35.878300+0000","last_became_active":"2024-04-25T19:11:29.936771+0000","last_became_peered":"2024-04-25T19:11:29.936771+0000","last_unstale":"2024-04-25T19:11:35.878300+0000","last_undegraded":"2024-04-25T19:11:35.878300+0000","last_fullsized":"2024-04-25T19:11:35.878300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:23.833133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171095+0000","last_change":"2024-04-25T19:11:29.944382+0000","last_active":"2024-04-25T19:11:37.171095+0000","last_peered":"2024-04-25T19:11:37.171095+0000","last_clean":"2024-04-25T19:11:37.171095+0000","last_became_active":"2024-04-25T19:11:29.938702+0000","last_became_peered":"2024-04-25T19:11:29.938702+0000","last_unstale":"2024-04-25T19:11:37.171095+0000","last_undegraded":"2024-04-25T19:11:37.171095+0000","last_fullsized":"2024-04-25T19:11:37.171095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990488+0000","last_change":"2024-04-25T19:11:29.934918+0000","last_active":"2024-04-25T19:11:36.990488+0000","last_peered":"2024-04-25T19:11:36.990488+0000","last_clean":"2024-04-25T19:11:36.990488+0000","last_became_active":"2024-04-25T19:11:29.934600+0000","last_became_peered":"2024-04-25T19:11:29.934600+0000","last_unstale":"2024-04-25T19:11:36.990488+0000","last_undegraded":"2024-04-25T19:11:36.990488+0000","last_fullsized":"2024-04-25T19:11:36.990488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:15.027554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962499+0000","last_change":"2024-04-25T19:11:29.944854+0000","last_active":"2024-04-25T19:11:34.962499+0000","last_peered":"2024-04-25T19:11:34.962499+0000","last_clean":"2024-04-25T19:11:34.962499+0000","last_became_active":"2024-04-25T19:11:29.944528+0000","last_became_peered":"2024-04-25T19:11:29.944528+0000","last_unstale":"2024-04-25T19:11:34.962499+0000","last_undegraded":"2024-04-25T19:11:34.962499+0000","last_fullsized":"2024-04-25T19:11:34.962499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963024+0000","last_change":"2024-04-25T19:11:29.944061+0000","last_active":"2024-04-25T19:11:34.963024+0000","last_peered":"2024-04-25T19:11:34.963024+0000","last_clean":"2024-04-25T19:11:34.963024+0000","last_became_active":"2024-04-25T19:11:29.943745+0000","last_became_peered":"2024-04-25T19:11:29.943745+0000","last_unstale":"2024-04-25T19:11:34.963024+0000","last_undegraded":"2024-04-25T19:11:34.963024+0000","last_fullsized":"2024-04-25T19:11:34.963024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172433+0000","last_change":"2024-04-25T19:11:29.944585+0000","last_active":"2024-04-25T19:11:37.172433+0000","last_peered":"2024-04-25T19:11:37.172433+0000","last_clean":"2024-04-25T19:11:37.172433+0000","last_became_active":"2024-04-25T19:11:29.944112+0000","last_became_peered":"2024-04-25T19:11:29.944112+0000","last_unstale":"2024-04-25T19:11:37.172433+0000","last_undegraded":"2024-04-25T19:11:37.172433+0000","last_fullsized":"2024-04-25T19:11:37.172433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:31.654149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170915+0000","last_change":"2024-04-25T19:11:29.949991+0000","last_active":"2024-04-25T19:11:37.170915+0000","last_peered":"2024-04-25T19:11:37.170915+0000","last_clean":"2024-04-25T19:11:37.170915+0000","last_became_active":"2024-04-25T19:11:29.948567+0000","last_became_peered":"2024-04-25T19:11:29.948567+0000","last_unstale":"2024-04-25T19:11:37.170915+0000","last_undegraded":"2024-04-25T19:11:37.170915+0000","last_fullsized":"2024-04-25T19:11:37.170915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171414+0000","last_change":"2024-04-25T19:11:29.930321+0000","last_active":"2024-04-25T19:11:37.171414+0000","last_peered":"2024-04-25T19:11:37.171414+0000","last_clean":"2024-04-25T19:11:37.171414+0000","last_became_active":"2024-04-25T19:11:29.929930+0000","last_became_peered":"2024-04-25T19:11:29.929930+0000","last_unstale":"2024-04-25T19:11:37.171414+0000","last_undegraded":"2024-04-25T19:11:37.171414+0000","last_fullsized":"2024-04-25T19:11:37.171414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963030+0000","last_change":"2024-04-25T19:11:29.944694+0000","last_active":"2024-04-25T19:11:34.963030+0000","last_peered":"2024-04-25T19:11:34.963030+0000","last_clean":"2024-04-25T19:11:34.963030+0000","last_became_active":"2024-04-25T19:11:29.944055+0000","last_became_peered":"2024-04-25T19:11:29.944055+0000","last_unstale":"2024-04-25T19:11:34.963030+0000","last_undegraded":"2024-04-25T19:11:34.963030+0000","last_fullsized":"2024-04-25T19:11:34.963030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171775+0000","last_change":"2024-04-25T19:11:29.931815+0000","last_active":"2024-04-25T19:11:37.171775+0000","last_peered":"2024-04-25T19:11:37.171775+0000","last_clean":"2024-04-25T19:11:37.171775+0000","last_became_active":"2024-04-25T19:11:29.931523+0000","last_became_peered":"2024-04-25T19:11:29.931523+0000","last_unstale":"2024-04-25T19:11:37.171775+0000","last_undegraded":"2024-04-25T19:11:37.171775+0000","last_fullsized":"2024-04-25T19:11:37.171775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:42.722556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962270+0000","last_change":"2024-04-25T19:11:29.951038+0000","last_active":"2024-04-25T19:11:34.962270+0000","last_peered":"2024-04-25T19:11:34.962270+0000","last_clean":"2024-04-25T19:11:34.962270+0000","last_became_active":"2024-04-25T19:11:29.950615+0000","last_became_peered":"2024-04-25T19:11:29.950615+0000","last_unstale":"2024-04-25T19:11:34.962270+0000","last_undegraded":"2024-04-25T19:11:34.962270+0000","last_fullsized":"2024-04-25T19:11:34.962270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:31.654149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878604+0000","last_change":"2024-04-25T19:11:29.932658+0000","last_active":"2024-04-25T19:11:35.878604+0000","last_peered":"2024-04-25T19:11:35.878604+0000","last_clean":"2024-04-25T19:11:35.878604+0000","last_became_active":"2024-04-25T19:11:29.932270+0000","last_became_peered":"2024-04-25T19:11:29.932270+0000","last_unstale":"2024-04-25T19:11:35.878604+0000","last_undegraded":"2024-04-25T19:11:35.878604+0000","last_fullsized":"2024-04-25T19:11:35.878604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470928+0000","last_change":"2024-04-25T19:11:29.929765+0000","last_active":"2024-04-25T19:11:35.470928+0000","last_peered":"2024-04-25T19:11:35.470928+0000","last_clean":"2024-04-25T19:11:35.470928+0000","last_became_active":"2024-04-25T19:11:29.929452+0000","last_became_peered":"2024-04-25T19:11:29.929452+0000","last_unstale":"2024-04-25T19:11:35.470928+0000","last_undegraded":"2024-04-25T19:11:35.470928+0000","last_fullsized":"2024-04-25T19:11:35.470928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172357+0000","last_change":"2024-04-25T19:11:29.936916+0000","last_active":"2024-04-25T19:11:37.172357+0000","last_peered":"2024-04-25T19:11:37.172357+0000","last_clean":"2024-04-25T19:11:37.172357+0000","last_became_active":"2024-04-25T19:11:29.935057+0000","last_became_peered":"2024-04-25T19:11:29.935057+0000","last_unstale":"2024-04-25T19:11:37.172357+0000","last_undegraded":"2024-04-25T19:11:37.172357+0000","last_fullsized":"2024-04-25T19:11:37.172357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:35.993870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:11:53.439 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T19:11:53.440 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T19:11:53.440 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T19:11:53.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T19:11:53.441 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T19:11:53.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.586+0000 7fa5d5506640 1 Processor -- start 2024-04-25T19:11:53.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.586+0000 7fa5d5506640 1 -- start start 2024-04-25T19:11:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5d5506640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa5d00b30e0 0x7fa5d00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5d5506640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5d009b540 0x7fa5d009b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5d5506640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d0161cf0 0x7fa5d0162100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5d5506640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d009be90 con 0x7fa5d009b540 2024-04-25T19:11:53.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5d5506640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d00b3650 con 0x7fa5d00b30e0 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5d5506640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d006c710 con 0x7fa5d0161cf0 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5ce7fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5d009b540 0x7fa5d009b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5ce7fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5d009b540 0x7fa5d009b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37144/0 (socket says 172.21.15.28:37144) 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5ce7fc640 1 -- 172.21.15.28:0/3596453551 learned_addr learned my addr 172.21.15.28:0/3596453551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5ceffd640 1 --2- 172.21.15.28:0/3596453551 >> v2:172.21.15.53:3300/0 conn(0x7fa5d00b30e0 0x7fa5d00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.587+0000 7fa5cf7fe640 1 --2- 172.21.15.28:0/3596453551 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d0161cf0 0x7fa5d0162100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5ce7fc640 1 -- 172.21.15.28:0/3596453551 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d0161cf0 msgr2=0x7fa5d0162100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5ce7fc640 1 --2- 172.21.15.28:0/3596453551 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d0161cf0 0x7fa5d0162100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5ce7fc640 1 -- 172.21.15.28:0/3596453551 >> v2:172.21.15.53:3300/0 conn(0x7fa5d00b30e0 msgr2=0x7fa5d00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5ce7fc640 1 --2- 172.21.15.28:0/3596453551 >> v2:172.21.15.53:3300/0 conn(0x7fa5d00b30e0 0x7fa5d00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5ce7fc640 1 -- 172.21.15.28:0/3596453551 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d006c9f0 con 0x7fa5d009b540 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5ce7fc640 1 --2- 172.21.15.28:0/3596453551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d009b540 0x7fa5d009b950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0072ad0 tx=0x7fa5c402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f4efeb1193527f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5cdffb640 1 -- 172.21.15.28:0/3596453551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4037070 con 0x7fa5d009b540 2024-04-25T19:11:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5cdffb640 1 -- 172.21.15.28:0/3596453551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c4032440 con 0x7fa5d009b540 2024-04-25T19:11:53.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.588+0000 7fa5cdffb640 1 -- 172.21.15.28:0/3596453551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4032810 con 0x7fa5d009b540 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 -- 172.21.15.28:0/3596453551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d009b540 msgr2=0x7fa5d009b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3596453551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d009b540 0x7fa5d009b950 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0072ad0 tx=0x7fa5c402a140 comp rx=0 tx=0).stop 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 -- 172.21.15.28:0/3596453551 shutdown_connections 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3596453551 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d0161cf0 0x7fa5d0162100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3596453551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d009b540 0x7fa5d009b950 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3596453551 >> v2:172.21.15.53:3300/0 conn(0x7fa5d00b30e0 0x7fa5d00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 -- 172.21.15.28:0/3596453551 >> 172.21.15.28:0/3596453551 conn(0x7fa5d00b7620 msgr2=0x7fa5d00b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 -- 172.21.15.28:0/3596453551 shutdown_connections 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.589+0000 7fa5d5506640 1 -- 172.21.15.28:0/3596453551 wait complete. 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.591+0000 7fa5d5506640 1 Processor -- start 2024-04-25T19:11:53.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.591+0000 7fa5d5506640 1 -- start start 2024-04-25T19:11:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.592+0000 7fa5d5506640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5d009b540 0x7fa5d00af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.592+0000 7fa5d5506640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d00b30e0 0x7fa5d01f9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.592+0000 7fa5d5506640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0161cf0 0x7fa5d01fa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.592+0000 7fa5d5506640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d00cfe90 con 0x7fa5d0161cf0 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.592+0000 7fa5d5506640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d00cfd10 con 0x7fa5d009b540 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.592+0000 7fa5d5506640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d0161940 con 0x7fa5d00b30e0 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0161cf0 0x7fa5d01fa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0161cf0 0x7fa5d01fa190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37152/0 (socket says 172.21.15.28:37152) 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 -- 172.21.15.28:0/3498668402 learned_addr learned my addr 172.21.15.28:0/3498668402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5ceffd640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5d009b540 0x7fa5d00af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5ce7fc640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d00b30e0 0x7fa5d01f9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 -- 172.21.15.28:0/3498668402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d00b30e0 msgr2=0x7fa5d01f9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d00b30e0 0x7fa5d01f9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 -- 172.21.15.28:0/3498668402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5d009b540 msgr2=0x7fa5d00af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5d009b540 0x7fa5d00af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 -- 172.21.15.28:0/3498668402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d00affd0 con 0x7fa5d0161cf0 2024-04-25T19:11:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5cf7fe640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0161cf0 0x7fa5d01fa190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8007e60 tx=0x7fa5c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5b77fe640 1 -- 172.21.15.28:0/3498668402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8014420 con 0x7fa5d0161cf0 2024-04-25T19:11:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5b77fe640 1 -- 172.21.15.28:0/3498668402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c8019070 con 0x7fa5d0161cf0 2024-04-25T19:11:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d00b02b0 con 0x7fa5d0161cf0 2024-04-25T19:11:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5b77fe640 1 -- 172.21.15.28:0/3498668402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8014820 con 0x7fa5d0161cf0 2024-04-25T19:11:53.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.593+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d00b0870 con 0x7fa5d0161cf0 2024-04-25T19:11:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.594+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d0050740 con 0x7fa5d0161cf0 2024-04-25T19:11:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.595+0000 7fa5b77fe640 1 -- 172.21.15.28:0/3498668402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa5c80149c0 con 0x7fa5d0161cf0 2024-04-25T19:11:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.596+0000 7fa5b77fe640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa5bc0744b0 0x7fa5bc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.596+0000 7fa5ceffd640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa5bc0744b0 0x7fa5bc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.596+0000 7fa5b77fe640 1 -- 172.21.15.28:0/3498668402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa5c8095c00 con 0x7fa5d0161cf0 2024-04-25T19:11:53.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.596+0000 7fa5ceffd640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa5bc0744b0 0x7fa5bc0769c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00b1f90 tx=0x7fa5c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.600+0000 7fa5b77fe640 1 -- 172.21.15.28:0/3498668402 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5c809c080 con 0x7fa5d0161cf0 2024-04-25T19:11:53.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.841+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa5d00afd60 con 0x7fa5d0161cf0 2024-04-25T19:11:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.844+0000 7fa5b77fe640 1 -- 172.21.15.28:0/3498668402 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+186743 (secure 0 0 0) 0x7fa5c805fa90 con 0x7fa5d0161cf0 2024-04-25T19:11:53.849 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa5bc0744b0 msgr2=0x7fa5bc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa5bc0744b0 0x7fa5bc0769c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa5d00b1f90 tx=0x7fa5c0008040 comp rx=0 tx=0).stop 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0161cf0 msgr2=0x7fa5d01fa190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0161cf0 0x7fa5d01fa190 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5c8007e60 tx=0x7fa5c800e4b0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 shutdown_connections 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa5bc0744b0 0x7fa5bc0769c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d0161cf0 0x7fa5d01fa190 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa5d00b30e0 0x7fa5d01f9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 --2- 172.21.15.28:0/3498668402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa5d009b540 0x7fa5d00af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 >> 172.21.15.28:0/3498668402 conn(0x7fa5d00b7620 msgr2=0x7fa5d00d1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 shutdown_connections 2024-04-25T19:11:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.847+0000 7fa5d5506640 1 -- 172.21.15.28:0/3498668402 wait complete. 2024-04-25T19:11:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2354405395},{"type":"v1","addr":"172.21.15.28:6833","nonce":2354405395}]},"active_addr":"172.21.15.28:6833/2354405395","active_change":"2024-04-25T19:11:08.826293+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4102,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3127490739}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":4120567146}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2340478905}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2276368979}]}]} 2024-04-25T19:11:53.864 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T19:11:53.864 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T19:11:53.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.975+0000 7ff982501640 1 Processor -- start 2024-04-25T19:11:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.975+0000 7ff982501640 1 -- start start 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff982501640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff97c0cfa00 0x7ff97c15e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff982501640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff97c0b21d0 0x7ff97c157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff982501640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c09b540 0x7ff97c09b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff982501640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7ff97c09be90 con 0x7ff97c0cfa00 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff982501640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7ff97c0cffe0 con 0x7ff97c0b21d0 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff982501640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff97c0d02c0 con 0x7ff97c09b540 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97bfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff97c0cfa00 0x7ff97c15e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97bfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff97c0cfa00 0x7ff97c15e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37156/0 (socket says 172.21.15.28:37156) 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97bfff640 1 -- 172.21.15.28:0/3014103282 learned_addr learned my addr 172.21.15.28:0/3014103282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97b7fe640 1 --2- 172.21.15.28:0/3014103282 >> v2:172.21.15.53:3300/0 conn(0x7ff97c0b21d0 0x7ff97c157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff980a9e640 1 --2- 172.21.15.28:0/3014103282 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c09b540 0x7ff97c09b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97bfff640 1 -- 172.21.15.28:0/3014103282 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c09b540 msgr2=0x7ff97c09b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97bfff640 1 --2- 172.21.15.28:0/3014103282 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c09b540 0x7ff97c09b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97bfff640 1 -- 172.21.15.28:0/3014103282 >> v2:172.21.15.53:3300/0 conn(0x7ff97c0b21d0 msgr2=0x7ff97c157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97bfff640 1 --2- 172.21.15.28:0/3014103282 >> v2:172.21.15.53:3300/0 conn(0x7ff97c0b21d0 0x7ff97c157570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.977+0000 7ff97bfff640 1 -- 172.21.15.28:0/3014103282 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97c15da20 con 0x7ff97c0cfa00 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.978+0000 7ff97bfff640 1 --2- 172.21.15.28:0/3014103282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0cfa00 0x7ff97c15e220 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff97c0d3980 tx=0x7ff96c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=836806deb7982a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.978+0000 7ff97affd640 1 -- 172.21.15.28:0/3014103282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c037070 con 0x7ff97c0cfa00 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.978+0000 7ff97affd640 1 -- 172.21.15.28:0/3014103282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff96c032440 con 0x7ff97c0cfa00 2024-04-25T19:11:53.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.978+0000 7ff97affd640 1 -- 172.21.15.28:0/3014103282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff96c032810 con 0x7ff97c0cfa00 2024-04-25T19:11:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 -- 172.21.15.28:0/3014103282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0cfa00 msgr2=0x7ff97c15e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 --2- 172.21.15.28:0/3014103282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0cfa00 0x7ff97c15e220 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff97c0d3980 tx=0x7ff96c02a140 comp rx=0 tx=0).stop 2024-04-25T19:11:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 -- 172.21.15.28:0/3014103282 shutdown_connections 2024-04-25T19:11:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 --2- 172.21.15.28:0/3014103282 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c09b540 0x7ff97c09b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 --2- 172.21.15.28:0/3014103282 >> v2:172.21.15.53:3300/0 conn(0x7ff97c0b21d0 0x7ff97c157570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 --2- 172.21.15.28:0/3014103282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0cfa00 0x7ff97c15e220 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 -- 172.21.15.28:0/3014103282 >> 172.21.15.28:0/3014103282 conn(0x7ff97c0b7620 msgr2=0x7ff97c0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:53.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 -- 172.21.15.28:0/3014103282 shutdown_connections 2024-04-25T19:11:53.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.979+0000 7ff982501640 1 -- 172.21.15.28:0/3014103282 wait complete. 2024-04-25T19:11:53.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.981+0000 7ff982501640 1 Processor -- start 2024-04-25T19:11:53.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.981+0000 7ff982501640 1 -- start start 2024-04-25T19:11:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff982501640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff97c09b540 0x7ff97c0d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff982501640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0b21d0 0x7ff97c1f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff982501640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c0cfa00 0x7ff97c144f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff982501640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c15d800 con 0x7ff97c0b21d0 2024-04-25T19:11:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff982501640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff97c15d680 con 0x7ff97c09b540 2024-04-25T19:11:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff982501640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff97c0b25e0 con 0x7ff97c0cfa00 2024-04-25T19:11:53.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff97b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0b21d0 0x7ff97c1f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff97b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0b21d0 0x7ff97c1f67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37160/0 (socket says 172.21.15.28:37160) 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff97b7fe640 1 -- 172.21.15.28:0/409390407 learned_addr learned my addr 172.21.15.28:0/409390407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff980a9e640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c0cfa00 0x7ff97c144f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.982+0000 7ff97bfff640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff97c09b540 0x7ff97c0d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff97b7fe640 1 -- 172.21.15.28:0/409390407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c0cfa00 msgr2=0x7ff97c144f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff97b7fe640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c0cfa00 0x7ff97c144f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff97b7fe640 1 -- 172.21.15.28:0/409390407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff97c09b540 msgr2=0x7ff97c0d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff97b7fe640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff97c09b540 0x7ff97c0d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff97b7fe640 1 -- 172.21.15.28:0/409390407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff97c0d21b0 con 0x7ff97c0b21d0 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff97b7fe640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0b21d0 0x7ff97c1f67a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff97000d620 tx=0x7ff97000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff978ff9640 1 -- 172.21.15.28:0/409390407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97000b620 con 0x7ff97c0b21d0 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff978ff9640 1 -- 172.21.15.28:0/409390407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff970019070 con 0x7ff97c0b21d0 2024-04-25T19:11:53.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff97c0d2490 con 0x7ff97c0b21d0 2024-04-25T19:11:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff978ff9640 1 -- 172.21.15.28:0/409390407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97000ba00 con 0x7ff97c0b21d0 2024-04-25T19:11:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.983+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff97c0d2a50 con 0x7ff97c0b21d0 2024-04-25T19:11:53.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.984+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff97c050740 con 0x7ff97c0b21d0 2024-04-25T19:11:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.985+0000 7ff978ff9640 1 -- 172.21.15.28:0/409390407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff97000bba0 con 0x7ff97c0b21d0 2024-04-25T19:11:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.985+0000 7ff978ff9640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9680744b0 0x7ff9680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.985+0000 7ff978ff9640 1 -- 172.21.15.28:0/409390407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff9700959f0 con 0x7ff97c0b21d0 2024-04-25T19:11:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.985+0000 7ff97bfff640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9680744b0 0x7ff9680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:53.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.986+0000 7ff97bfff640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9680744b0 0x7ff9680769c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff96c034000 tx=0x7ff96c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:53.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:53.990+0000 7ff978ff9640 1 -- 172.21.15.28:0/409390407 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff97005fe40 con 0x7ff97c0b21d0 2024-04-25T19:11:54.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.178+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff97c0d1f40 con 0x7ff97c0b21d0 2024-04-25T19:11:54.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.179+0000 7ff978ff9640 1 -- 172.21.15.28:0/409390407 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7ff97005f740 con 0x7ff97c0b21d0 2024-04-25T19:11:54.181 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:54.181 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9680744b0 msgr2=0x7ff9680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9680744b0 0x7ff9680769c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff96c034000 tx=0x7ff96c02e000 comp rx=0 tx=0).stop 2024-04-25T19:11:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0b21d0 msgr2=0x7ff97c1f67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0b21d0 0x7ff97c1f67a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff97000d620 tx=0x7ff97000dae0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 shutdown_connections 2024-04-25T19:11:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9680744b0 0x7ff9680769c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff97c0cfa00 0x7ff97c144f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff97c0b21d0 0x7ff97c1f67a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 --2- 172.21.15.28:0/409390407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff97c09b540 0x7ff97c0d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 >> 172.21.15.28:0/409390407 conn(0x7ff97c0b7620 msgr2=0x7ff97c0b1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 shutdown_connections 2024-04-25T19:11:54.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.181+0000 7ff982501640 1 -- 172.21.15.28:0/409390407 wait complete. 2024-04-25T19:11:54.194 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T19:11:54.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:54.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.305+0000 7f23d3892640 1 Processor -- start 2024-04-25T19:11:54.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.305+0000 7f23d3892640 1 -- start start 2024-04-25T19:11:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d3892640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f23cc0b5130 0x7f23cc161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d3892640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23cc06ded0 0x7f23cc06e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d3892640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23cc06e820 con 0x7f23cc0d0830 2024-04-25T19:11:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d3892640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f23cc163c30 con 0x7f23cc06ded0 2024-04-25T19:11:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d3892640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f23cc164a90 con 0x7f23cc0b5130 2024-04-25T19:11:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d0e2d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23cc06ded0 0x7f23cc06e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d0e2d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23cc06ded0 0x7f23cc06e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37162/0 (socket says 172.21.15.28:37162) 2024-04-25T19:11:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d0e2d640 1 -- 172.21.15.28:0/2043854458 learned_addr learned my addr 172.21.15.28:0/2043854458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d162e640 1 --2- 172.21.15.28:0/2043854458 >> v2:172.21.15.53:3300/0 conn(0x7f23cc0b5130 0x7f23cc161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d0e2d640 1 -- 172.21.15.28:0/2043854458 >> v2:172.21.15.53:3300/0 conn(0x7f23cc0b5130 msgr2=0x7f23cc161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d0e2d640 1 --2- 172.21.15.28:0/2043854458 >> v2:172.21.15.53:3300/0 conn(0x7f23cc0b5130 0x7f23cc161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d0e2d640 1 -- 172.21.15.28:0/2043854458 >> v1:172.21.15.53:6789/0 conn(0x7f23cc0d0830 legacy=0x7f23cc0d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.307+0000 7f23d0e2d640 1 -- 172.21.15.28:0/2043854458 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23cc163e80 con 0x7f23cc06ded0 2024-04-25T19:11:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.308+0000 7f23d0e2d640 1 --2- 172.21.15.28:0/2043854458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc06ded0 0x7f23cc06e2e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f23cc06ec40 tx=0x7f23c002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b78b447b52fd64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.308+0000 7f23cbfff640 1 -- 172.21.15.28:0/2043854458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c0037070 con 0x7f23cc06ded0 2024-04-25T19:11:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.308+0000 7f23cbfff640 1 -- 172.21.15.28:0/2043854458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23c0032440 con 0x7f23cc06ded0 2024-04-25T19:11:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.308+0000 7f23cbfff640 1 -- 172.21.15.28:0/2043854458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c0032810 con 0x7f23cc06ded0 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.309+0000 7f23d3892640 1 -- 172.21.15.28:0/2043854458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc06ded0 msgr2=0x7f23cc06e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.309+0000 7f23d3892640 1 --2- 172.21.15.28:0/2043854458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc06ded0 0x7f23cc06e2e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f23cc06ec40 tx=0x7f23c002a230 comp rx=0 tx=0).stop 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.309+0000 7f23d3892640 1 -- 172.21.15.28:0/2043854458 shutdown_connections 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.309+0000 7f23d3892640 1 --2- 172.21.15.28:0/2043854458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc06ded0 0x7f23cc06e2e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.309+0000 7f23d3892640 1 --2- 172.21.15.28:0/2043854458 >> v2:172.21.15.53:3300/0 conn(0x7f23cc0b5130 0x7f23cc161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.309+0000 7f23d3892640 1 -- 172.21.15.28:0/2043854458 >> 172.21.15.28:0/2043854458 conn(0x7f23cc0b7ad0 msgr2=0x7f23cc0b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.309+0000 7f23d3892640 1 -- 172.21.15.28:0/2043854458 shutdown_connections 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.309+0000 7f23d3892640 1 -- 172.21.15.28:0/2043854458 wait complete. 2024-04-25T19:11:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.311+0000 7f23d3892640 1 Processor -- start 2024-04-25T19:11:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.311+0000 7f23d3892640 1 -- start start 2024-04-25T19:11:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d3892640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23cc06ded0 0x7f23cc14de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d3892640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23cc0b5130 0x7f23cc1eb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d3892640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc0d0830 0x7f23cc1ebcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d3892640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc164720 con 0x7f23cc0d0830 2024-04-25T19:11:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d3892640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f23cc164440 con 0x7f23cc0b5130 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d3892640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f23cc1607e0 con 0x7f23cc06ded0 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d1e2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc0d0830 0x7f23cc1ebcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d1e2f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc0d0830 0x7f23cc1ebcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37166/0 (socket says 172.21.15.28:37166) 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d1e2f640 1 -- 172.21.15.28:0/4177256123 learned_addr learned my addr 172.21.15.28:0/4177256123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d0e2d640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23cc0b5130 0x7f23cc1eb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.312+0000 7f23d162e640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23cc06ded0 0x7f23cc14de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23d1e2f640 1 -- 172.21.15.28:0/4177256123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23cc06ded0 msgr2=0x7f23cc14de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23d1e2f640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23cc06ded0 0x7f23cc14de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23d1e2f640 1 -- 172.21.15.28:0/4177256123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23cc0b5130 msgr2=0x7f23cc1eb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23d1e2f640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23cc0b5130 0x7f23cc1eb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23d1e2f640 1 -- 172.21.15.28:0/4177256123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23cc0c9930 con 0x7f23cc0d0830 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23d1e2f640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc0d0830 0x7f23cc1ebcb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00b970 tx=0x7f23bc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23c9ffb640 1 -- 172.21.15.28:0/4177256123 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc019070 con 0x7f23cc0d0830 2024-04-25T19:11:54.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23c9ffb640 1 -- 172.21.15.28:0/4177256123 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23bc00cb40 con 0x7f23cc0d0830 2024-04-25T19:11:54.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23cc0c9c10 con 0x7f23cc0d0830 2024-04-25T19:11:54.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23c9ffb640 1 -- 172.21.15.28:0/4177256123 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23bc014440 con 0x7f23cc0d0830 2024-04-25T19:11:54.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.313+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23cc0ca1d0 con 0x7f23cc0d0830 2024-04-25T19:11:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.314+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23cc1ec270 con 0x7f23cc0d0830 2024-04-25T19:11:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.315+0000 7f23c9ffb640 1 -- 172.21.15.28:0/4177256123 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f23bc01c070 con 0x7f23cc0d0830 2024-04-25T19:11:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.315+0000 7f23c9ffb640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23b80744b0 0x7f23b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.315+0000 7f23c9ffb640 1 -- 172.21.15.28:0/4177256123 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f23bc0950c0 con 0x7f23cc0d0830 2024-04-25T19:11:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.315+0000 7f23d162e640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23b80744b0 0x7f23b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.316+0000 7f23d162e640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23b80744b0 0x7f23b80769c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23cc06ec10 tx=0x7f23c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.320+0000 7f23c9ffb640 1 -- 172.21.15.28:0/4177256123 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f23bc05ef40 con 0x7f23cc0d0830 2024-04-25T19:11:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.505+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23cc050740 con 0x7f23cc0d0830 2024-04-25T19:11:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.506+0000 7f23c9ffb640 1 -- 172.21.15.28:0/4177256123 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f23bc063240 con 0x7f23cc0d0830 2024-04-25T19:11:54.508 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:54.508 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23b80744b0 msgr2=0x7f23b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23b80744b0 0x7f23b80769c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23cc06ec10 tx=0x7f23c4008040 comp rx=0 tx=0).stop 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc0d0830 msgr2=0x7f23cc1ebcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc0d0830 0x7f23cc1ebcb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f23bc00b970 tx=0x7f23bc00be30 comp rx=0 tx=0).stop 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 shutdown_connections 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23b80744b0 0x7f23b80769c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23cc0d0830 0x7f23cc1ebcb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23cc0b5130 0x7f23cc1eb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 --2- 172.21.15.28:0/4177256123 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f23cc06ded0 0x7f23cc14de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 >> 172.21.15.28:0/4177256123 conn(0x7f23cc0b7ad0 msgr2=0x7f23cc0b9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 shutdown_connections 2024-04-25T19:11:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.509+0000 7f23d3892640 1 -- 172.21.15.28:0/4177256123 wait complete. 2024-04-25T19:11:54.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T19:11:54.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T19:11:54.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T19:11:54.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T19:11:54.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T19:11:54.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T19:11:54.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T19:11:54.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T19:11:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.741+0000 7f3dbbfff640 1 Processor -- start 2024-04-25T19:11:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.741+0000 7f3dbbfff640 1 -- start start 2024-04-25T19:11:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.741+0000 7f82d6cdd640 1 Processor -- start 2024-04-25T19:11:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.742+0000 7f82d6cdd640 1 -- start start 2024-04-25T19:11:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.742+0000 7f51216f9640 1 Processor -- start 2024-04-25T19:11:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.742+0000 7f51216f9640 1 -- start start 2024-04-25T19:11:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.743+0000 7f3dbbfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3db40551a0 0x7f3db4054230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.743+0000 7f3dbbfff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3db40e04d0 0x7f3db4039450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.743+0000 7f3dbbfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40089e0 0x7f3db4008df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.743+0000 7f3dbbfff640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f3db4008550 con 0x7f3db40551a0 2024-04-25T19:11:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.743+0000 7f3dbbfff640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f3db40e08e0 con 0x7f3db40e04d0 2024-04-25T19:11:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.743+0000 7f3dbbfff640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f3db4055940 con 0x7f3db40089e0 2024-04-25T19:11:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dbaffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3db40551a0 0x7f3db4054230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dbb7fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40089e0 0x7f3db4008df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dbaffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3db40551a0 0x7f3db4054230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37176/0 (socket says 172.21.15.28:37176) 2024-04-25T19:11:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dbaffd640 1 -- 172.21.15.28:0/3560387280 learned_addr learned my addr 172.21.15.28:0/3560387280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dbb7fe640 1 --2- 172.21.15.28:0/3560387280 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40089e0 0x7f3db4008df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47004/0 (socket says 172.21.15.28:47004) 2024-04-25T19:11:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dba7fc640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3db40e04d0 0x7f3db4039450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f82d6cdd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82c804f340 0x7f82c8033a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f82d6cdd640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c804f750 con 0x7f82c8033fb0 2024-04-25T19:11:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f82d6cdd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82c80343c0 con 0x7f82c8052fb0 2024-04-25T19:11:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f82d6cdd640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f82c80533c0 con 0x7f82c804f340 2024-04-25T19:11:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f82d54da640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82c804f340 0x7f82c8033a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f82d54da640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82c804f340 0x7f82c8033a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37178/0 (socket says 172.21.15.28:37178) 2024-04-25T19:11:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f82d54da640 1 -- 172.21.15.28:0/1156782210 learned_addr learned my addr 172.21.15.28:0/1156782210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f1b3d7be640 1 Processor -- start 2024-04-25T19:11:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dba7fc640 1 -- 172.21.15.28:0/3560387280 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40089e0 msgr2=0x7f3db4008df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dba7fc640 1 --2- 172.21.15.28:0/3560387280 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40089e0 0x7f3db4008df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dba7fc640 1 -- 172.21.15.28:0/3560387280 >> v2:172.21.15.28:3300/0 conn(0x7f3db40551a0 msgr2=0x7f3db4054230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dba7fc640 1 --2- 172.21.15.28:0/3560387280 >> v2:172.21.15.28:3300/0 conn(0x7f3db40551a0 0x7f3db4054230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f3dba7fc640 1 -- 172.21.15.28:0/3560387280 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db4007f40 con 0x7f3db40e04d0 2024-04-25T19:11:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.745+0000 7f3dbaffd640 1 --2- 172.21.15.28:0/3560387280 >> v2:172.21.15.28:3300/0 conn(0x7f3db40551a0 0x7f3db4054230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.745+0000 7f8321fb2640 1 Processor -- start 2024-04-25T19:11:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f1b3d7be640 1 -- start start 2024-04-25T19:11:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.745+0000 7f511affd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5114038330 0x7f5114053690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58910/0 (socket says 172.21.15.28:58910) 2024-04-25T19:11:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f51216f9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f51140485e0 0x7f5114053da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f51216f9640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51140532a0 con 0x7f5114038330 2024-04-25T19:11:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f51216f9640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51140e7fe0 con 0x7f5114006f00 2024-04-25T19:11:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f51216f9640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5114034850 con 0x7f51140485e0 2024-04-25T19:11:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f511b7fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5114006f00 0x7f5114007310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40222/0 (socket says 172.21.15.28:40222) 2024-04-25T19:11:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f511b7fe640 1 -- 172.21.15.28:0/1659126839 learned_addr learned my addr 172.21.15.28:0/1659126839 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f511a7fc640 1 --2- 172.21.15.28:0/1659126839 >> v2:172.21.15.53:3300/0 conn(0x7f51140485e0 0x7f5114053da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f3dba7fc640 1 --2- 172.21.15.28:0/3560387280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40e04d0 0x7f3db4039450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009a00 tx=0x7f3db0029fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a9d94693480fb14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f5119ffb640 1 -- 172.21.15.28:0/1659126839 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 685272994 0 0) 0x7f51140e7fe0 con 0x7f5114006f00 2024-04-25T19:11:54.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f5119ffb640 1 -- 172.21.15.28:0/1659126839 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5104003520 con 0x7f5114006f00 2024-04-25T19:11:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f3db9ffb640 1 -- 172.21.15.28:0/3560387280 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0037070 con 0x7f3db40e04d0 2024-04-25T19:11:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f3db9ffb640 1 -- 172.21.15.28:0/3560387280 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3db002ad10 con 0x7f3db40e04d0 2024-04-25T19:11:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f3db9ffb640 1 -- 172.21.15.28:0/3560387280 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db0032660 con 0x7f3db40e04d0 2024-04-25T19:11:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f1b3d7be640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 0x7f1b300e4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 0x7f1b300e4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 0x7f1b300e4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47020/0 (socket says 172.21.15.28:47020) 2024-04-25T19:11:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f1b37fff640 1 -- 172.21.15.28:0/3213413573 learned_addr learned my addr 172.21.15.28:0/3213413573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f1b3d7be640 1 -- 172.21.15.28:0/3213413573 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b30055870 con 0x7f1b300e0660 2024-04-25T19:11:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f1b3d7be640 1 -- 172.21.15.28:0/3213413573 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b300e4880 con 0x7f1b30050d40 2024-04-25T19:11:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f1b3d7be640 1 -- 172.21.15.28:0/3213413573 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f1b300e4650 con 0x7f1b30055460 2024-04-25T19:11:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.745+0000 7f8321fb2640 1 -- start start 2024-04-25T19:11:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f8321fb2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8314050600 0x7f83140345e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f8321fb2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054140 0x7f8314054550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f8321fb2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f831404fce0 con 0x7f8314054a90 2024-04-25T19:11:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f83217b1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8314054a90 0x7f8314006470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58942/0 (socket says 172.21.15.28:58942) 2024-04-25T19:11:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f83217b1640 1 -- 172.21.15.28:0/3974861489 learned_addr learned my addr 172.21.15.28:0/3974861489 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f8321fb2640 1 -- 172.21.15.28:0/3974861489 --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8314048ea0 con 0x7f8314050600 2024-04-25T19:11:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f8321fb2640 1 -- 172.21.15.28:0/3974861489 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8314049040 con 0x7f8314054140 2024-04-25T19:11:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f831bfff640 1 --2- 172.21.15.28:0/3974861489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054140 0x7f8314054550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f511a7fc640 1 -- 172.21.15.28:0/1659126839 >> v1:172.21.15.53:6789/0 conn(0x7f5114006f00 legacy=0x7f5114007310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f511a7fc640 1 -- 172.21.15.28:0/1659126839 >> v1:172.21.15.28:6789/0 conn(0x7f5114038330 legacy=0x7f5114053690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f511a7fc640 1 -- 172.21.15.28:0/1659126839 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5114050370 con 0x7f51140485e0 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f511a7fc640 1 --2- 172.21.15.28:0/1659126839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51140485e0 0x7f5114053da0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f51100097a0 tx=0x7f511002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b78b35efcada30d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f511affd640 1 --1- 172.21.15.28:0/1659126839 >> v1:172.21.15.28:6789/0 conn(0x7f5114038330 0x7f5114053690 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f5119ffb640 1 -- 172.21.15.28:0/1659126839 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110037070 con 0x7f51140485e0 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f3dbbfff640 1 -- 172.21.15.28:0/3560387280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40e04d0 msgr2=0x7f3db4039450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f3dbbfff640 1 --2- 172.21.15.28:0/3560387280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40e04d0 0x7f3db4039450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009a00 tx=0x7f3db0029fc0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f3dbbfff640 1 -- 172.21.15.28:0/3560387280 shutdown_connections 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f3dbbfff640 1 --2- 172.21.15.28:0/3560387280 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40089e0 0x7f3db4008df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f3dbbfff640 1 --2- 172.21.15.28:0/3560387280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40e04d0 0x7f3db4039450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f3dbbfff640 1 --2- 172.21.15.28:0/3560387280 >> v2:172.21.15.28:3300/0 conn(0x7f3db40551a0 0x7f3db4054230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f3dbbfff640 1 -- 172.21.15.28:0/3560387280 >> 172.21.15.28:0/3560387280 conn(0x7f3db4055e00 msgr2=0x7f3db4056210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f0f00185640 1 Processor -- start 2024-04-25T19:11:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f0f00185640 1 -- start start 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.745+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1957783449 0 0) 0x7f82c804f750 con 0x7f82c8033fb0 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.745+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82bc003520 con 0x7f82c8033fb0 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3190360992 0 0) 0x7f82bc003520 con 0x7f82c8033fb0 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 >> v2:172.21.15.28:3300/0 conn(0x7f82c804f340 msgr2=0x7f82c8033a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 --2- 172.21.15.28:0/1156782210 >> v2:172.21.15.28:3300/0 conn(0x7f82c804f340 0x7f82c8033a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 >> v1:172.21.15.53:6789/0 conn(0x7f82c8052fb0 legacy=0x7f82c8007320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c8007a30 con 0x7f82c8033fb0 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f82c0001670 con 0x7f82c8033fb0 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 >> v1:172.21.15.28:6789/0 conn(0x7f82c8033fb0 legacy=0x7f82c8038340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82bc003a00 0x7f82bc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82bc004800 0x7f82bc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82bc0099b0 0x7f82bc00bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82c80533c0 con 0x7f82bc0099b0 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f82bc00c3c0 con 0x7f82bc004800 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f82bc00c5c0 con 0x7f82bc003a00 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d5cdb640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82bc003a00 0x7f82bc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.746+0000 7f82d64dc640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82bc0099b0 0x7f82bc00bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f82d64dc640 1 -- 172.21.15.28:0/1156782210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82bc003a00 msgr2=0x7f82bc003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f82d64dc640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82bc003a00 0x7f82bc003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f82d64dc640 1 -- 172.21.15.28:0/1156782210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82bc004800 msgr2=0x7f82bc004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f82d64dc640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82bc004800 0x7f82bc004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f82d64dc640 1 -- 172.21.15.28:0/1156782210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82bc003520 con 0x7f82bc0099b0 2024-04-25T19:11:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f5119ffb640 1 -- 172.21.15.28:0/1659126839 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5110032440 con 0x7f51140485e0 2024-04-25T19:11:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f5119ffb640 1 -- 172.21.15.28:0/1659126839 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110032810 con 0x7f51140485e0 2024-04-25T19:11:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f1b36ffd640 1 -- 172.21.15.28:0/3213413573 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 604872611 0 0) 0x7f1b300e4880 con 0x7f1b30050d40 2024-04-25T19:11:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f1b37fff640 1 -- 172.21.15.28:0/3213413573 >> v1:172.21.15.53:6789/0 conn(0x7f1b30050d40 legacy=0x7f1b30051150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f1b37fff640 1 -- 172.21.15.28:0/3213413573 >> v1:172.21.15.28:6789/0 conn(0x7f1b300e0660 legacy=0x7f1b300e0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f1b37fff640 1 -- 172.21.15.28:0/3213413573 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b300e4b60 con 0x7f1b30055460 2024-04-25T19:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f3dbbfff640 1 -- 172.21.15.28:0/3560387280 shutdown_connections 2024-04-25T19:11:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f8320fb0640 1 --2- 172.21.15.28:0/3974861489 >> v2:172.21.15.28:3300/0 conn(0x7f8314050600 0x7f83140345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f0f00185640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0ef80d21d0 0x7f0ef806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f0f00185640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ef80b1280 0x7f0ef806cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f0f00185640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef80af4b0 con 0x7f0ef8164840 2024-04-25T19:11:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f0f00185640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f0ef80af280 con 0x7f0ef80b1280 2024-04-25T19:11:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f0f00185640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f0ef806df60 con 0x7f0ef80d21d0 2024-04-25T19:11:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f82d64dc640 1 -- 172.21.15.28:0/1156782210 reap_dead start 2024-04-25T19:11:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f82d64dc640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82bc0099b0 0x7f82bc00bdd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f82c0007b70 tx=0x7f82bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde85f114dd43f8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f51216f9640 1 -- 172.21.15.28:0/1659126839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51140485e0 msgr2=0x7f5114053da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f51216f9640 1 --2- 172.21.15.28:0/1659126839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51140485e0 0x7f5114053da0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f51100097a0 tx=0x7f511002a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f0efdf21640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0ef8164840 0x7f0ef80d1ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40240/0 (socket says 172.21.15.28:40240) 2024-04-25T19:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efdf21640 1 -- 172.21.15.28:0/1193944588 learned_addr learned my addr 172.21.15.28:0/1193944588 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efe722640 1 --2- 172.21.15.28:0/1193944588 >> v2:172.21.15.28:3300/0 conn(0x7f0ef80b1280 0x7f0ef806cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efd720640 1 --2- 172.21.15.28:0/1193944588 >> v2:172.21.15.53:3300/0 conn(0x7f0ef80d21d0 0x7f0ef806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efcf1f640 1 -- 172.21.15.28:0/1193944588 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2992064453 0 0) 0x7f0ef80af4b0 con 0x7f0ef8164840 2024-04-25T19:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efcf1f640 1 -- 172.21.15.28:0/1193944588 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ee4003520 con 0x7f0ef8164840 2024-04-25T19:11:54.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0007bb0 con 0x7f82bc0099b0 2024-04-25T19:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f51216f9640 1 -- 172.21.15.28:0/1659126839 shutdown_connections 2024-04-25T19:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f51216f9640 1 --2- 172.21.15.28:0/1659126839 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51140485e0 0x7f5114053da0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f51216f9640 1 -- 172.21.15.28:0/1659126839 >> 172.21.15.28:0/1659126839 conn(0x7f5114055170 msgr2=0x7f5114055580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f51216f9640 1 -- 172.21.15.28:0/1659126839 shutdown_connections 2024-04-25T19:11:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efe722640 1 -- 172.21.15.28:0/1193944588 >> v2:172.21.15.53:3300/0 conn(0x7f0ef80d21d0 msgr2=0x7f0ef806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efe722640 1 --2- 172.21.15.28:0/1193944588 >> v2:172.21.15.53:3300/0 conn(0x7f0ef80d21d0 0x7f0ef806c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efe722640 1 -- 172.21.15.28:0/1193944588 >> v1:172.21.15.53:6789/0 conn(0x7f0ef8164840 legacy=0x7f0ef80d1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efe722640 1 -- 172.21.15.28:0/1193944588 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef80af790 con 0x7f0ef80b1280 2024-04-25T19:11:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f0efd720640 1 --2- 172.21.15.28:0/1193944588 >> v2:172.21.15.53:3300/0 conn(0x7f0ef80d21d0 0x7f0ef806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c0053620 con 0x7f82bc0099b0 2024-04-25T19:11:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f1b37fff640 1 --2- 172.21.15.28:0/3213413573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 0x7f1b300e4110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0098a0 tx=0x7f1b2c02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bce3c2792674055 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f82d4cd9640 1 -- 172.21.15.28:0/1156782210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0053a10 con 0x7f82bc0099b0 2024-04-25T19:11:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.744+0000 7f22fbd57640 1 Processor -- start 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.745+0000 7f22fbd57640 1 -- start start 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f22fbd57640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22f40cf9e0 0x7f22f40cfdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f22f9af3640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f22f4162330 0x7f22f41683d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40234/0 (socket says 172.21.15.28:40234) 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f22f9af3640 1 -- 172.21.15.28:0/161850122 learned_addr learned my addr 172.21.15.28:0/161850122 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f22f92f2640 1 --2- 172.21.15.28:0/161850122 >> v2:172.21.15.53:3300/0 conn(0x7f22f40cf9e0 0x7f22f40cfdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f22fbd57640 1 -- 172.21.15.28:0/161850122 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22f4162740 con 0x7f22f4165370 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f22fbd57640 1 -- 172.21.15.28:0/161850122 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22f409b6e0 con 0x7f22f4162330 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.747+0000 7f22fbd57640 1 -- 172.21.15.28:0/161850122 --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f22f409b9c0 con 0x7f22f40cf9e0 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f22f8af1640 1 -- 172.21.15.28:0/161850122 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1111731577 0 0) 0x7f22f409b6e0 con 0x7f22f4162330 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f22f8af1640 1 -- 172.21.15.28:0/161850122 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e0003520 con 0x7f22f4162330 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f22f92f2640 1 -- 172.21.15.28:0/161850122 >> v1:172.21.15.53:6789/0 conn(0x7f22f4162330 legacy=0x7f22f41683d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f22f92f2640 1 -- 172.21.15.28:0/161850122 >> v1:172.21.15.28:6789/0 conn(0x7f22f4165370 legacy=0x7f22f4165780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.748+0000 7f22f92f2640 1 -- 172.21.15.28:0/161850122 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f409bca0 con 0x7f22f40cf9e0 2024-04-25T19:11:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f3dbbfff640 1 -- 172.21.15.28:0/3560387280 wait complete. 2024-04-25T19:11:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.751+0000 7f1b36ffd640 1 -- 172.21.15.28:0/3213413573 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c037070 con 0x7f1b30055460 2024-04-25T19:11:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.751+0000 7f1b36ffd640 1 -- 172.21.15.28:0/3213413573 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2c032440 con 0x7f1b30055460 2024-04-25T19:11:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.751+0000 7f1b36ffd640 1 -- 172.21.15.28:0/3213413573 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c032810 con 0x7f1b30055460 2024-04-25T19:11:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f51216f9640 1 -- 172.21.15.28:0/1659126839 wait complete. 2024-04-25T19:11:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.751+0000 7f22f92f2640 1 --2- 172.21.15.28:0/161850122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22f40cf9e0 0x7f22f40cfdf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f22f00097a0 tx=0x7f22f002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90b9d25a6c5d7da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f831bfff640 1 -- 172.21.15.28:0/3974861489 >> v2:172.21.15.28:3300/0 conn(0x7f8314050600 msgr2=0x7f83140345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.749+0000 7f831bfff640 1 --2- 172.21.15.28:0/3974861489 >> v2:172.21.15.28:3300/0 conn(0x7f8314050600 0x7f83140345e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f831bfff640 1 -- 172.21.15.28:0/3974861489 >> v1:172.21.15.28:6789/0 conn(0x7f8314054a90 legacy=0x7f8314006470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.750+0000 7f831bfff640 1 -- 172.21.15.28:0/3974861489 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8314048bc0 con 0x7f8314054140 2024-04-25T19:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.751+0000 7f831bfff640 1 --2- 172.21.15.28:0/3974861489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054140 0x7f8314054550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f83080097a0 tx=0x7f830802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e71dbe82071956c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.751+0000 7f22f8af1640 1 -- 172.21.15.28:0/161850122 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0037070 con 0x7f22f40cf9e0 2024-04-25T19:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.751+0000 7f22f8af1640 1 -- 172.21.15.28:0/161850122 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f0032440 con 0x7f22f40cf9e0 2024-04-25T19:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.751+0000 7f22f8af1640 1 -- 172.21.15.28:0/161850122 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0032850 con 0x7f22f40cf9e0 2024-04-25T19:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f82d6cdd640 1 -- 172.21.15.28:0/1156782210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82bc0099b0 msgr2=0x7f82bc00bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f82d6cdd640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82bc0099b0 0x7f82bc00bdd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f82c0007b70 tx=0x7f82bc005830 comp rx=0 tx=0).stop 2024-04-25T19:11:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f82d6cdd640 1 -- 172.21.15.28:0/1156782210 shutdown_connections 2024-04-25T19:11:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f82d6cdd640 1 --2- 172.21.15.28:0/1156782210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82bc0099b0 0x7f82bc00bdd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f82d6cdd640 1 -- 172.21.15.28:0/1156782210 >> 172.21.15.28:0/1156782210 conn(0x7f82c8054cf0 msgr2=0x7f82c8055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f1b3d7be640 1 -- 172.21.15.28:0/3213413573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 msgr2=0x7f1b300e4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f1b3d7be640 1 --2- 172.21.15.28:0/3213413573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 0x7f1b300e4110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0098a0 tx=0x7f1b2c02a250 comp rx=0 tx=0).stop 2024-04-25T19:11:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f0efe722640 1 --2- 172.21.15.28:0/1193944588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef80b1280 0x7f0ef806cab0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0ef80b4aa0 tx=0x7f0ef002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6cd17a46a95d71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f1b3d7be640 1 -- 172.21.15.28:0/3213413573 shutdown_connections 2024-04-25T19:11:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f1b3d7be640 1 --2- 172.21.15.28:0/3213413573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 0x7f1b300e4110 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f1b3d7be640 1 -- 172.21.15.28:0/3213413573 >> 172.21.15.28:0/3213413573 conn(0x7f1b30055e00 msgr2=0x7f1b30056210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f1b3d7be640 1 -- 172.21.15.28:0/3213413573 shutdown_connections 2024-04-25T19:11:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f82d6cdd640 1 -- 172.21.15.28:0/1156782210 shutdown_connections 2024-04-25T19:11:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f3dbbfff640 1 Processor -- start 2024-04-25T19:11:54.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f0efcf1f640 1 -- 172.21.15.28:0/1193944588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0007740 con 0x7f0ef80b1280 2024-04-25T19:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f22fbd57640 1 -- 172.21.15.28:0/161850122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22f40cf9e0 msgr2=0x7f22f40cfdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f3dbbfff640 1 -- start start 2024-04-25T19:11:54.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f22fbd57640 1 --2- 172.21.15.28:0/161850122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22f40cf9e0 0x7f22f40cfdf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f22f00097a0 tx=0x7f22f002a250 comp rx=0 tx=0).stop 2024-04-25T19:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f831b7fe640 1 -- 172.21.15.28:0/3974861489 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8308037070 con 0x7f8314054140 2024-04-25T19:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.753+0000 7f831b7fe640 1 -- 172.21.15.28:0/3974861489 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8308032440 con 0x7f8314054140 2024-04-25T19:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.753+0000 7f831b7fe640 1 -- 172.21.15.28:0/3974861489 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f830802ae60 con 0x7f8314054140 2024-04-25T19:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f1b3d7be640 1 -- 172.21.15.28:0/3213413573 wait complete. 2024-04-25T19:11:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f82d6cdd640 1 -- 172.21.15.28:0/1156782210 wait complete. 2024-04-25T19:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.753+0000 7f51216f9640 1 Processor -- start 2024-04-25T19:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f0efcf1f640 1 -- 172.21.15.28:0/1193944588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef00078e0 con 0x7f0ef80b1280 2024-04-25T19:11:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.752+0000 7f0efcf1f640 1 -- 172.21.15.28:0/1193944588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0007cb0 con 0x7f0ef80b1280 2024-04-25T19:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.753+0000 7f51216f9640 1 -- start start 2024-04-25T19:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.753+0000 7f5cf5527640 1 Processor -- start 2024-04-25T19:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.753+0000 7f5cf5527640 1 -- start start 2024-04-25T19:11:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f0f00185640 1 -- 172.21.15.28:0/1193944588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef80b1280 msgr2=0x7f0ef806cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f0f00185640 1 --2- 172.21.15.28:0/1193944588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef80b1280 0x7f0ef806cab0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0ef80b4aa0 tx=0x7f0ef002a6a0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f8321fb2640 1 -- 172.21.15.28:0/3974861489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054140 msgr2=0x7f8314054550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f0f00185640 1 -- 172.21.15.28:0/1193944588 shutdown_connections 2024-04-25T19:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f0f00185640 1 --2- 172.21.15.28:0/1193944588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef80b1280 0x7f0ef806cab0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f0f00185640 1 --2- 172.21.15.28:0/1193944588 >> v2:172.21.15.53:3300/0 conn(0x7f0ef80d21d0 0x7f0ef806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f0f00185640 1 -- 172.21.15.28:0/1193944588 >> 172.21.15.28:0/1193944588 conn(0x7f0ef80739e0 msgr2=0x7f0ef80b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f8321fb2640 1 --2- 172.21.15.28:0/3974861489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054140 0x7f8314054550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f83080097a0 tx=0x7f830802a110 comp rx=0 tx=0).stop 2024-04-25T19:11:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f0f00185640 1 -- 172.21.15.28:0/1193944588 shutdown_connections 2024-04-25T19:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f3dbbfff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40089e0 0x7f3db403b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f3dbbfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40e04d0 0x7f3db4177dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40d2590 0x7f3db4039a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbbfff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3db4009330 con 0x7f3db40d2590 2024-04-25T19:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbbfff640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3db40081e0 con 0x7f3db40089e0 2024-04-25T19:11:54.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbbfff640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f3db40094b0 con 0x7f3db40e04d0 2024-04-25T19:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f22fbd57640 1 -- 172.21.15.28:0/161850122 shutdown_connections 2024-04-25T19:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f22fbd57640 1 --2- 172.21.15.28:0/161850122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22f40cf9e0 0x7f22f40cfdf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f22fbd57640 1 -- 172.21.15.28:0/161850122 >> 172.21.15.28:0/161850122 conn(0x7f22f40b7620 msgr2=0x7f22f40b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40d2590 0x7f3db4039a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40d2590 0x7f3db4039a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37212/0 (socket says 172.21.15.28:37212) 2024-04-25T19:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbb7fe640 1 -- 172.21.15.28:0/1629406693 learned_addr learned my addr 172.21.15.28:0/1629406693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbaffd640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40089e0 0x7f3db403b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f1b3d7be640 1 Processor -- start 2024-04-25T19:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cf5527640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5cf009b540 0x7f5cf015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cf5527640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5cf015d320 0x7f5cf006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cf5527640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5cf006cab0 0x7f5cf015da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cee7fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5cf015d320 0x7f5cf006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.754+0000 7f0f00185640 1 -- 172.21.15.28:0/1193944588 wait complete. 2024-04-25T19:11:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f8321fb2640 1 -- 172.21.15.28:0/3974861489 shutdown_connections 2024-04-25T19:11:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f8321fb2640 1 --2- 172.21.15.28:0/3974861489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054140 0x7f8314054550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f8321fb2640 1 --2- 172.21.15.28:0/3974861489 >> v2:172.21.15.28:3300/0 conn(0x7f8314050600 0x7f83140345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f8321fb2640 1 -- 172.21.15.28:0/3974861489 >> 172.21.15.28:0/3974861489 conn(0x7f831400aa70 msgr2=0x7f831400ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbaffd640 1 -- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40e04d0 msgr2=0x7f3db4177dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T19:11:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbaffd640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3db40e04d0 0x7f3db4177dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbaffd640 1 -- 172.21.15.28:0/1629406693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40d2590 msgr2=0x7f3db4039a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f3dbaffd640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40d2590 0x7f3db4039a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f3dbaffd640 1 -- 172.21.15.28:0/1629406693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db40ddd80 con 0x7f3db40089e0 2024-04-25T19:11:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f8321fb2640 1 -- 172.21.15.28:0/3974861489 shutdown_connections 2024-04-25T19:11:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f3dbb7fe640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3db40d2590 0x7f3db4039a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f82d6cdd640 1 Processor -- start 2024-04-25T19:11:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cef7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5cf006cab0 0x7f5cf015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cf5527640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf00d2740 con 0x7f5cf006cab0 2024-04-25T19:11:54.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cf5527640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf006cec0 con 0x7f5cf009b540 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cf5527640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf00d1b30 con 0x7f5cf015d320 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cef7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5cf006cab0 0x7f5cf015da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37218/0 (socket says 172.21.15.28:37218) 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f5cef7fe640 1 -- 172.21.15.28:0/3745818170 learned_addr learned my addr 172.21.15.28:0/3745818170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f5ceeffd640 1 --2- 172.21.15.28:0/3745818170 >> v2:172.21.15.53:3300/0 conn(0x7f5cf009b540 0x7f5cf015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f5cef7fe640 1 -- 172.21.15.28:0/3745818170 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5cf015d320 msgr2=0x7f5cf006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f5cef7fe640 1 --2- 172.21.15.28:0/3745818170 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5cf015d320 0x7f5cf006c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f5cef7fe640 1 -- 172.21.15.28:0/3745818170 >> v2:172.21.15.53:3300/0 conn(0x7f5cf009b540 msgr2=0x7f5cf015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f5cef7fe640 1 --2- 172.21.15.28:0/3745818170 >> v2:172.21.15.53:3300/0 conn(0x7f5cf009b540 0x7f5cf015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f5cef7fe640 1 -- 172.21.15.28:0/3745818170 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf00d1e10 con 0x7f5cf006cab0 2024-04-25T19:11:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f5ceeffd640 1 --2- 172.21.15.28:0/3745818170 >> v2:172.21.15.53:3300/0 conn(0x7f5cf009b540 0x7f5cf015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f82d6cdd640 1 -- start start 2024-04-25T19:11:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.755+0000 7f1b3d7be640 1 -- start start 2024-04-25T19:11:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f8321fb2640 1 -- 172.21.15.28:0/3974861489 wait complete. 2024-04-25T19:11:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f51216f9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5114006f00 0x7f51140df3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f51216f9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5114038330 0x7f511404c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.756+0000 7f3dbaffd640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40089e0 0x7f3db403b920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00e8b0 tx=0x7f3dac00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f51216f9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51140485e0 0x7f511404e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f511affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5114006f00 0x7f51140df3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f22fbd57640 1 -- 172.21.15.28:0/161850122 shutdown_connections 2024-04-25T19:11:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f511affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5114006f00 0x7f51140df3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53322/0 (socket says 172.21.15.28:53322) 2024-04-25T19:11:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f511a7fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5114038330 0x7f511404c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f511a7fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5114038330 0x7f511404c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47040/0 (socket says 172.21.15.28:47040) 2024-04-25T19:11:54.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac004030 con 0x7f3db40089e0 2024-04-25T19:11:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f0f00185640 1 Processor -- start 2024-04-25T19:11:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f0f00185640 1 -- start start 2024-04-25T19:11:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db40de060 con 0x7f3db40089e0 2024-04-25T19:11:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f511affd640 1 -- 172.21.15.28:0/930640984 learned_addr learned my addr 172.21.15.28:0/930640984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f51216f9640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51140542e0 con 0x7f51140485e0 2024-04-25T19:11:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f51216f9640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5114006700 con 0x7f5114006f00 2024-04-25T19:11:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db40de620 con 0x7f3db40089e0 2024-04-25T19:11:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f51216f9640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5114054460 con 0x7f5114038330 2024-04-25T19:11:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dac009370 con 0x7f3db40089e0 2024-04-25T19:11:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac004030 con 0x7f3db40089e0 2024-04-25T19:11:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.757+0000 7f5cef7fe640 1 --2- 172.21.15.28:0/3745818170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 0x7f5cf015da60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00733a0 tx=0x7f5ce802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68dc215474b193bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b3d7be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b30050d40 0x7f1b3004a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b3d7be640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 0x7f1b300e3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b3d7be640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b300e0660 0x7f1b3004c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b3d7be640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1b30008100 con 0x7f1b30050d40 2024-04-25T19:11:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b3d7be640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1b30007eb0 con 0x7f1b300e0660 2024-04-25T19:11:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b3d7be640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f1b30009630 con 0x7f1b30055460 2024-04-25T19:11:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b30050d40 0x7f1b3004a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b30050d40 0x7f1b3004a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37222/0 (socket says 172.21.15.28:37222) 2024-04-25T19:11:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b37fff640 1 -- 172.21.15.28:0/1848767358 learned_addr learned my addr 172.21.15.28:0/1848767358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f5cedffb640 1 -- 172.21.15.28:0/3745818170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8036070 con 0x7f5cf006cab0 2024-04-25T19:11:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f5cedffb640 1 -- 172.21.15.28:0/3745818170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce8031600 con 0x7f5cf006cab0 2024-04-25T19:11:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f5cedffb640 1 -- 172.21.15.28:0/3745818170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80319d0 con 0x7f5cf006cab0 2024-04-25T19:11:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d9c001030 con 0x7f3db40089e0 2024-04-25T19:11:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f511affd640 1 -- 172.21.15.28:0/930640984 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5114038330 msgr2=0x7f511404c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f511affd640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5114038330 0x7f511404c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f511affd640 1 -- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51140485e0 msgr2=0x7f511404e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f511affd640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51140485e0 0x7f511404e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f511affd640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f511404daf0 con 0x7f5114006f00 2024-04-25T19:11:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b37fff640 1 -- 172.21.15.28:0/1848767358 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 msgr2=0x7f1b300e3990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T19:11:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b37fff640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1b30055460 0x7f1b300e3990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b37fff640 1 -- 172.21.15.28:0/1848767358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b300e0660 msgr2=0x7f1b3004c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f8321fb2640 1 Processor -- start 2024-04-25T19:11:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d6cdd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82bc0099b0 0x7f82c8044d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b3cfbd640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b300e0660 0x7f1b3004c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f8321fb2640 1 -- start start 2024-04-25T19:11:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b37fff640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1b300e0660 0x7f1b3004c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d6cdd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c80e5b00 0x7f82c8180b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f1b37fff640 1 -- 172.21.15.28:0/1848767358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b300499d0 con 0x7f1b30050d40 2024-04-25T19:11:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0f00185640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef80b1280 0x7f0ef815ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d5cdb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82bc0099b0 0x7f82c8044d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.758+0000 7f22fbd57640 1 -- 172.21.15.28:0/161850122 wait complete. 2024-04-25T19:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0f00185640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ef80d21d0 0x7f0ef814cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d5cdb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82bc0099b0 0x7f82c8044d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53336/0 (socket says 172.21.15.28:53336) 2024-04-25T19:11:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d5cdb640 1 -- 172.21.15.28:0/682331221 learned_addr learned my addr 172.21.15.28:0/682331221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0f00185640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8164840 0x7f0ef814d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d6cdd640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82c8170830 0x7f82c8170c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0f00185640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef8164c50 con 0x7f0ef80b1280 2024-04-25T19:11:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d6cdd640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f82c8006560 con 0x7f82c80e5b00 2024-04-25T19:11:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0f00185640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef80afd50 con 0x7f0ef8164840 2024-04-25T19:11:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d6cdd640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f82c8006280 con 0x7f82bc0099b0 2024-04-25T19:11:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d64dc640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82c8170830 0x7f82c8170c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f1b37fff640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b30050d40 0x7f1b3004a470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0097f0 tx=0x7f1b2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0efe722640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8164840 0x7f0ef814d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0efe722640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8164840 0x7f0ef814d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53352/0 (socket says 172.21.15.28:53352) 2024-04-25T19:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0efe722640 1 -- 172.21.15.28:0/3964247260 learned_addr learned my addr 172.21.15.28:0/3964247260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef80d2910 con 0x7f0ef80d21d0 2024-04-25T19:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f0efd720640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ef80d21d0 0x7f0ef814cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f511affd640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5114006f00 0x7f51140df3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f510c00b410 tx=0x7f510c00b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f82d6cdd640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f82c8006840 con 0x7f82c8170830 2024-04-25T19:11:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f5cf5527640 1 -- 172.21.15.28:0/3745818170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 msgr2=0x7f5cf015da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f5cf5527640 1 --2- 172.21.15.28:0/3745818170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 0x7f5cf015da60 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00733a0 tx=0x7f5ce802a610 comp rx=0 tx=0).stop 2024-04-25T19:11:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f5cf5527640 1 -- 172.21.15.28:0/3745818170 shutdown_connections 2024-04-25T19:11:54.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f5cf5527640 1 --2- 172.21.15.28:0/3745818170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 0x7f5cf015da60 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f5cf5527640 1 --2- 172.21.15.28:0/3745818170 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5cf015d320 0x7f5cf006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f0efdf21640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef80b1280 0x7f0ef815ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f0efe722640 1 -- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ef80d21d0 msgr2=0x7f0ef814cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f5cf5527640 1 --2- 172.21.15.28:0/3745818170 >> v2:172.21.15.53:3300/0 conn(0x7f5cf009b540 0x7f5cf015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f5cf5527640 1 -- 172.21.15.28:0/3745818170 >> 172.21.15.28:0/3745818170 conn(0x7f5cf00739e0 msgr2=0x7f5cf00b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f0efe722640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ef80d21d0 0x7f0ef814cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f0efe722640 1 -- 172.21.15.28:0/3964247260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef80b1280 msgr2=0x7f0ef815ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f82d64dc640 1 -- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82bc0099b0 msgr2=0x7f82c8044d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f82d64dc640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82bc0099b0 0x7f82c8044d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f82d64dc640 1 -- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c80e5b00 msgr2=0x7f82c8180b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f0efe722640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ef80b1280 0x7f0ef815ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f0efe722640 1 -- 172.21.15.28:0/3964247260 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef815eb90 con 0x7f0ef8164840 2024-04-25T19:11:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c015020 con 0x7f5114006f00 2024-04-25T19:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3dac0041d0 con 0x7f3db40089e0 2024-04-25T19:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c041070 con 0x7f1b30050d40 2024-04-25T19:11:54.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f510c011830 con 0x7f5114006f00 2024-04-25T19:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2c03c930 con 0x7f1b30050d40 2024-04-25T19:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f0efe722640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8164840 0x7f0ef814d4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009eb0 tx=0x7f0ef0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510c011ca0 con 0x7f5114006f00 2024-04-25T19:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c037040 con 0x7f1b30050d40 2024-04-25T19:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f5cf5527640 1 -- 172.21.15.28:0/3745818170 shutdown_connections 2024-04-25T19:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.759+0000 7f82d54da640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c80e5b00 0x7f82c8180b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f82d64dc640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c80e5b00 0x7f82c8180b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f82d64dc640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c8171180 con 0x7f82c8170830 2024-04-25T19:11:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.760+0000 7f82d54da640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82c80e5b00 0x7f82c8180b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3004f020 con 0x7f1b30050d40 2024-04-25T19:11:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f3da37fe640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3da80748e0 0x7f3da8076df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8321fb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8314050600 0x7f83140475b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3dac00f040 con 0x7f3db40089e0 2024-04-25T19:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8321fb2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8314054140 0x7f8314045360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8321fb2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054a90 0x7f83140458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f3da37fe640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7f3da807e020 0x7f3da80804b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8321fb2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8314048970 con 0x7f8314050600 2024-04-25T19:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8321fb2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8314034390 con 0x7f8314054140 2024-04-25T19:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8321fb2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8314007b90 con 0x7f8314054a90 2024-04-25T19:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8320fb0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8314050600 0x7f83140475b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8320fb0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8314050600 0x7f83140475b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37242/0 (socket says 172.21.15.28:37242) 2024-04-25T19:11:54.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f8320fb0640 1 -- 172.21.15.28:0/3789913455 learned_addr learned my addr 172.21.15.28:0/3789913455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3dac01ee70 con 0x7f3da807e020 2024-04-25T19:11:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3dac095710 con 0x7f3db40089e0 2024-04-25T19:11:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f831bfff640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8314054140 0x7f8314045360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f83217b1640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054a90 0x7f83140458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3004cac0 con 0x7f1b30050d40 2024-04-25T19:11:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f22fbd57640 1 Processor -- start 2024-04-25T19:11:54.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f8320fb0640 1 -- 172.21.15.28:0/3789913455 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054a90 msgr2=0x7f83140458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f8320fb0640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8314054a90 0x7f83140458a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f8320fb0640 1 -- 172.21.15.28:0/3789913455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8314054140 msgr2=0x7f8314045360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f8320fb0640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8314054140 0x7f8314045360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f8320fb0640 1 -- 172.21.15.28:0/3789913455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8314172520 con 0x7f8314050600 2024-04-25T19:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f51216f9640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f511404ca50 con 0x7f5114006f00 2024-04-25T19:11:54.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f51216f9640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f511404d010 con 0x7f5114006f00 2024-04-25T19:11:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.761+0000 7f82d64dc640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82c8170830 0x7f82c8170c40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82c8051630 tx=0x7f82c0001970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f3dbb7fe640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7f3da807e020 0x7f3da80804b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f8320fb0640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8314050600 0x7f83140475b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f831000d800 tx=0x7f831000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.763+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310005990 con 0x7f8314050600 2024-04-25T19:11:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.763+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8310005b30 con 0x7f8314050600 2024-04-25T19:11:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.763+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8310005d20 con 0x7f8314050600 2024-04-25T19:11:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.763+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8314044c70 con 0x7f8314050600 2024-04-25T19:11:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.763+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83141727e0 con 0x7f8314050600 2024-04-25T19:11:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f51140df930 con 0x7f5114006f00 2024-04-25T19:11:54.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1b18001030 con 0x7f1b30050d40 2024-04-25T19:11:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef003f070 con 0x7f0ef8164840 2024-04-25T19:11:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef0031bc0 con 0x7f0ef8164840 2024-04-25T19:11:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0009450 con 0x7f0ef8164840 2024-04-25T19:11:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.763+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef815e5d0 con 0x7f0ef8164840 2024-04-25T19:11:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.763+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef814af90 con 0x7f0ef8164840 2024-04-25T19:11:54.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0064070 con 0x7f82c8170830 2024-04-25T19:11:54.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f3dba7fc640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3da80748e0 0x7f3da8076df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1b2c044070 con 0x7f1b30050d40 2024-04-25T19:11:54.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f510c018020 con 0x7f5114006f00 2024-04-25T19:11:54.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f51037fe640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5108074580 0x7f5108076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f831403b220 con 0x7f8314050600 2024-04-25T19:11:54.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c0005070 con 0x7f82c8170830 2024-04-25T19:11:54.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0007bb0 con 0x7f82c8170830 2024-04-25T19:11:54.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f82d6cdd640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c8181070 con 0x7f82c8170830 2024-04-25T19:11:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f82d6cdd640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c8045910 con 0x7f82c8170830 2024-04-25T19:11:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f510c03ae00 con 0x7f5114006f00 2024-04-25T19:11:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f51037fe640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7f510807dd40 0x7f51080801d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f511a7fc640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5108074580 0x7f5108076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51080808a0 con 0x7f510807dd40 2024-04-25T19:11:54.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f510c0959d0 con 0x7f5114006f00 2024-04-25T19:11:54.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f3dbb7fe640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7f3da807e020 0x7f3da80804b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.762+0000 7f5cf5527640 1 -- 172.21.15.28:0/3745818170 wait complete. 2024-04-25T19:11:54.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f5cf5527640 1 Processor -- start 2024-04-25T19:11:54.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f3dba7fc640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3da80748e0 0x7f3da8076df0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3db0034000 tx=0x7f3db002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.764+0000 7f22fbd57640 1 -- start start 2024-04-25T19:11:54.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f511a7fc640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5108074580 0x7f5108076a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5110034000 tx=0x7f511002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f1b34ff9640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1b1c074580 0x7f1b1c076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0ef8050740 con 0x7f0ef8164840 2024-04-25T19:11:54.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.766+0000 7f1b377fe640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1b1c074580 0x7f1b1c076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.766+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 <== osd.4 v2:172.21.15.53:6824/2055146181 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3dac01ee70 con 0x7f3da807e020 2024-04-25T19:11:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.766+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f831001b020 con 0x7f8314050600 2024-04-25T19:11:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.766+0000 7f82d6cdd640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f82c80054a0 con 0x7f82c8170830 2024-04-25T19:11:54.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.766+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1b2c0b6110 con 0x7f1b30050d40 2024-04-25T19:11:54.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f83197fa640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83000745b0 0x7f8300076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f1b34ff9640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f1b1c07dd40 0x7f1b1c0801d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f1b377fe640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1b1c074580 0x7f1b1c076a90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1b24000f60 tx=0x7f1b240079b0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8310095120 con 0x7f8314050600 2024-04-25T19:11:54.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b2c041e70 con 0x7f1b1c07dd40 2024-04-25T19:11:54.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1b2c0bd050 con 0x7f1b30050d40 2024-04-25T19:11:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f1b3cfbd640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f1b1c07dd40 0x7f1b1c0801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f831bfff640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83000745b0 0x7f8300076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f83197fa640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7f830007dd00 0x7f8300080190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8300080860 con 0x7f830007dd00 2024-04-25T19:11:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f831009c050 con 0x7f8314050600 2024-04-25T19:11:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.766+0000 7f511b7fe640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7f510807dd40 0x7f51080801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f511b7fe640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7f510807dd40 0x7f51080801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f82c0067070 con 0x7f82c8170830 2024-04-25T19:11:54.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f83217b1640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7f830007dd00 0x7f8300080190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f831bfff640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83000745b0 0x7f8300076ac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8308034000 tx=0x7f830802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f1b3cfbd640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f1b1c07dd40 0x7f1b1c0801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.765+0000 7f5cf5527640 1 -- start start 2024-04-25T19:11:54.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f82c67fc640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82b40745b0 0x7f82b4076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22fbd57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22f40cf9e0 0x7f22f40a0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f83217b1640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7f830007dd00 0x7f8300080190 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1b2c041e70 con 0x7f1b1c07dd40 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22fbd57640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22f4162330 0x7f22f40b0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22fbd57640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22f4165370 0x7f22f40b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22fbd57640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22f4168ae0 con 0x7f22f40cf9e0 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22fbd57640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f22f40d0330 con 0x7f22f4165370 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22fbd57640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f22f40d1b30 con 0x7f22f4162330 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22f9af3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22f40cf9e0 0x7f22f40a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22f9af3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22f40cf9e0 0x7f22f40a0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37250/0 (socket says 172.21.15.28:37250) 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f22f9af3640 1 -- 172.21.15.28:0/2807250931 learned_addr learned my addr 172.21.15.28:0/2807250931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f22f92f2640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22f4162330 0x7f22f40b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f22f9af3640 1 -- 172.21.15.28:0/2807250931 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22f4162330 msgr2=0x7f22f40b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f22f9af3640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f22f4162330 0x7f22f40b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f22f9af3640 1 -- 172.21.15.28:0/2807250931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22f4165370 msgr2=0x7f22f40b0ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T19:11:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f22f9af3640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22f4165370 0x7f22f40b0ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f22f9af3640 1 -- 172.21.15.28:0/2807250931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f4142070 con 0x7f22f40cf9e0 2024-04-25T19:11:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f51080808a0 con 0x7f510807dd40 2024-04-25T19:11:54.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.767+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0ef0065a80 con 0x7f0ef8164840 2024-04-25T19:11:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f82c00d6bc0 con 0x7f82c8170830 2024-04-25T19:11:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f82c67fc640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f82b407dd00 0x7f82b4080190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f82c0063ec0 con 0x7f82b407dd00 2024-04-25T19:11:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f82d5cdb640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82b40745b0 0x7f82b4076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f82d54da640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f82b407dd00 0x7f82b4080190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f82c00dd050 con 0x7f82c8170830 2024-04-25T19:11:54.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f5cf5527640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 0x7f5cf0154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.768+0000 7f5cf5527640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf009b540 0x7f5cf00af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f22f9af3640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22f40cf9e0 0x7f22f40a0050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00cbc0 tx=0x7f22ec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 <== osd.2 v2:172.21.15.28:6816/3410596961 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8300080860 con 0x7f830007dd00 2024-04-25T19:11:54.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f0eee7fc640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0edc074580 0x7f0edc076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec017070 con 0x7f22f40cf9e0 2024-04-25T19:11:54.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0ef00b6ea0 con 0x7f0ef8164840 2024-04-25T19:11:54.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.770+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22ec00ad40 con 0x7f22f40cf9e0 2024-04-25T19:11:54.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.770+0000 7f22fbd57640 1 -- 172.21.15.28:0/2807250931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f4142350 con 0x7f22f40cf9e0 2024-04-25T19:11:54.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f0eee7fc640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f0edc07dd50 0x7f0edc0801e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0edc0808b0 con 0x7f0edc07dd50 2024-04-25T19:11:54.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.770+0000 7f22fbd57640 1 -- 172.21.15.28:0/2807250931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f4142910 con 0x7f22f40cf9e0 2024-04-25T19:11:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0ef00b7290 con 0x7f0ef8164840 2024-04-25T19:11:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.770+0000 7f82d5cdb640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82b40745b0 0x7f82b4076ac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f82cc005030 tx=0x7f82cc007cc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.770+0000 7f0efdf21640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0edc074580 0x7f0edc076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f0efdf21640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0edc074580 0x7f0edc076a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ef40011e0 tx=0x7f0ef4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f0efd720640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f0edc07dd50 0x7f0edc0801e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec012620 con 0x7f22f40cf9e0 2024-04-25T19:11:54.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f22fbd57640 1 -- 172.21.15.28:0/2807250931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f22dc001030 con 0x7f22f40cf9e0 2024-04-25T19:11:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f5ceeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 0x7f5cf0154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f5cf5527640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5cf015d320 0x7f5cf00afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f5cf5527640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf015d730 con 0x7f5cf006cab0 2024-04-25T19:11:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f5cf5527640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00d23d0 con 0x7f5cf009b540 2024-04-25T19:11:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.769+0000 7f5cf5527640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf01546b0 con 0x7f5cf015d320 2024-04-25T19:11:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.770+0000 7f5ceeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 0x7f5cf0154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37252/0 (socket says 172.21.15.28:37252) 2024-04-25T19:11:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.770+0000 7f5ceeffd640 1 -- 172.21.15.28:0/3636179041 learned_addr learned my addr 172.21.15.28:0/3636179041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f5ceeffd640 1 -- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5cf015d320 msgr2=0x7f5cf00afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f5ceeffd640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5cf015d320 0x7f5cf00afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f5ceeffd640 1 -- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf009b540 msgr2=0x7f5cf00af8b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f5ceeffd640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5cf009b540 0x7f5cf00af8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f5ceeffd640 1 -- 172.21.15.28:0/3636179041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf00b0580 con 0x7f5cf006cab0 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.771+0000 7f5ceeffd640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 0x7f5cf0154170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c820 tx=0x7f5ce000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.772+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0015070 con 0x7f5cf006cab0 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.772+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf00b0860 con 0x7f5cf006cab0 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.772+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce0007830 con 0x7f5cf006cab0 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.772+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0015070 con 0x7f5cf006cab0 2024-04-25T19:11:54.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.772+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf02033f0 con 0x7f5cf006cab0 2024-04-25T19:11:54.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.772+0000 7f82d54da640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f82b407dd00 0x7f82b4080190 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.773+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5cd0001030 con 0x7f5cf006cab0 2024-04-25T19:11:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.773+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f22ec01a070 con 0x7f22f40cf9e0 2024-04-25T19:11:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.773+0000 7f22ea7fc640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22d80744e0 0x7f22d80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.774+0000 7f22f92f2640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22d80744e0 0x7f22d80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.775+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f22ec095400 con 0x7f22f40cf9e0 2024-04-25T19:11:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.775+0000 7f22f92f2640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22d80744e0 0x7f22d80769f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f22f0034000 tx=0x7f22f002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.775+0000 7f22ea7fc640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f22d807dca0 0x7f22d8080130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.775+0000 7f0efd720640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f0edc07dd50 0x7f0edc0801e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.775+0000 7f22fa2f4640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f22d807dca0 0x7f22d8080130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.776+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22ec016ec0 con 0x7f22d807dca0 2024-04-25T19:11:54.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.775+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5ce0018070 con 0x7f5cf006cab0 2024-04-25T19:11:54.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.775+0000 7f5cd77fe640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cdc0746b0 0x7f5cdc076bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.776+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f22ec09c050 con 0x7f22f40cf9e0 2024-04-25T19:11:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.776+0000 7f5cee7fc640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cdc0746b0 0x7f5cdc076bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.776+0000 7f22fa2f4640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f22d807dca0 0x7f22d8080130 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.777+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 <== osd.5 v2:172.21.15.53:6800/2342659583 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0edc0808b0 con 0x7f0edc07dd50 2024-04-25T19:11:54.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.777+0000 7f5cee7fc640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cdc0746b0 0x7f5cdc076bc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5cf015e5a0 tx=0x7f5ce4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.777+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 <== osd.1 v2:172.21.15.28:6808/1898594658 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22ec016ec0 con 0x7f22d807dca0 2024-04-25T19:11:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.773+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f82c0063ec0 con 0x7f82b407dd00 2024-04-25T19:11:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.777+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5ce005b910 con 0x7f5cf006cab0 2024-04-25T19:11:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.777+0000 7f5cd77fe640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f5cdc07e900 0x7f5cdc080df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:54.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.777+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cdc0814c0 con 0x7f5cdc07e900 2024-04-25T19:11:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.778+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5ce005bb10 con 0x7f5cf006cab0 2024-04-25T19:11:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.779+0000 7f5cef7fe640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f5cdc07e900 0x7f5cdc080df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.779+0000 7f5cef7fe640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f5cdc07e900 0x7f5cdc080df0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.780+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 <== osd.7 v2:172.21.15.53:6808/79451145 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5cdc0814c0 con 0x7f5cdc07e900 2024-04-25T19:11:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.799+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1b18002dd0 con 0x7f1b1c07dd40 2024-04-25T19:11:54.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.800+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3d9c002dd0 con 0x7f3da807e020 2024-04-25T19:11:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.800+0000 7f1b34ff9640 1 -- 172.21.15.28:0/1848767358 <== osd.0 v2:172.21.15.28:6800/1137456103 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1b18002dd0 con 0x7f1b1c07dd40 2024-04-25T19:11:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.800+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5114005920 con 0x7f510807dd40 2024-04-25T19:11:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f1b1c07dd40 msgr2=0x7f1b1c0801d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3d7be640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f1b1c07dd40 0x7f1b1c0801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1b1c074580 msgr2=0x7f1b1c076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3d7be640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1b1c074580 0x7f1b1c076a90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1b24000f60 tx=0x7f1b240079b0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b30050d40 msgr2=0x7f1b3004a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3d7be640 1 --2- 172.21.15.28:0/1848767358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b30050d40 0x7f1b3004a470 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0097f0 tx=0x7f1b2c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3cfbd640 1 -- 172.21.15.28:0/1848767358 reap_dead start 2024-04-25T19:11:54.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f51037fe640 1 -- 172.21.15.28:0/930640984 <== osd.3 v2:172.21.15.28:6818/1928233777 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5114005920 con 0x7f510807dd40 2024-04-25T19:11:54.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8314045190 con 0x7f830007dd00 2024-04-25T19:11:54.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f3da37fe640 1 -- 172.21.15.28:0/1629406693 <== osd.4 v2:172.21.15.53:6824/2055146181 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3d9c002dd0 con 0x7f3da807e020 2024-04-25T19:11:54.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 shutdown_connections 2024-04-25T19:11:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.801+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 >> 172.21.15.28:0/1848767358 conn(0x7f1b30055e00 msgr2=0x7f1b300de270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7f510807dd40 msgr2=0x7f51080801d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7f510807dd40 0x7f51080801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 shutdown_connections 2024-04-25T19:11:54.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f1b3d7be640 1 -- 172.21.15.28:0/1848767358 wait complete. 2024-04-25T19:11:54.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f83197fa640 1 -- 172.21.15.28:0/3789913455 <== osd.2 v2:172.21.15.28:6816/3410596961 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8314045190 con 0x7f830007dd00 2024-04-25T19:11:54.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5108074580 msgr2=0x7f5108076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5108074580 0x7f5108076a90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5110034000 tx=0x7f511002e000 comp rx=0 tx=0).stop 2024-04-25T19:11:54.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5114006f00 msgr2=0x7f51140df3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 --2- 172.21.15.28:0/930640984 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5114006f00 0x7f51140df3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f510c00b410 tx=0x7f510c00b8d0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f511b7fe640 1 -- 172.21.15.28:0/930640984 reap_dead start 2024-04-25T19:11:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 shutdown_connections 2024-04-25T19:11:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 >> 172.21.15.28:0/930640984 conn(0x7f5114055170 msgr2=0x7f51140399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7f830007dd00 msgr2=0x7f8300080190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f8321fb2640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7f830007dd00 0x7f8300080190 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 shutdown_connections 2024-04-25T19:11:54.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7f3da807e020 msgr2=0x7f3da80804b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.802+0000 7f3dbbfff640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7f3da807e020 0x7f3da80804b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f51017fa640 1 -- 172.21.15.28:0/930640984 wait complete. 2024-04-25T19:11:54.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3da80748e0 msgr2=0x7f3da8076df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f3dbbfff640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3da80748e0 0x7f3da8076df0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3db0034000 tx=0x7f3db002e000 comp rx=0 tx=0).stop 2024-04-25T19:11:54.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83000745b0 msgr2=0x7f8300076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f8321fb2640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83000745b0 0x7f8300076ac0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8308034000 tx=0x7f830802e000 comp rx=0 tx=0).stop 2024-04-25T19:11:54.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8314050600 msgr2=0x7f83140475b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f8321fb2640 1 --2- 172.21.15.28:0/3789913455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8314050600 0x7f83140475b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f831000d800 tx=0x7f831000dcc0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f83217b1640 1 -- 172.21.15.28:0/3789913455 reap_dead start 2024-04-25T19:11:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40089e0 msgr2=0x7f3db403b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f3dbbfff640 1 --2- 172.21.15.28:0/1629406693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3db40089e0 0x7f3db403b920 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00e8b0 tx=0x7f3dac00ed70 comp rx=0 tx=0).stop 2024-04-25T19:11:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f3dbb7fe640 1 -- 172.21.15.28:0/1629406693 reap_dead start 2024-04-25T19:11:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 shutdown_connections 2024-04-25T19:11:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 >> 172.21.15.28:0/1629406693 conn(0x7f3db4055e00 msgr2=0x7f3db40d6670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 shutdown_connections 2024-04-25T19:11:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.803+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 >> 172.21.15.28:0/3789913455 conn(0x7f831400aa70 msgr2=0x7f831404c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.804+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 shutdown_connections 2024-04-25T19:11:54.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.804+0000 7f3dbbfff640 1 -- 172.21.15.28:0/1629406693 wait complete. 2024-04-25T19:11:54.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.804+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 shutdown_connections 2024-04-25T19:11:54.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.804+0000 7f8321fb2640 1 -- 172.21.15.28:0/3789913455 wait complete. 2024-04-25T19:11:54.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.806+0000 7f82d6cdd640 1 -- 172.21.15.28:0/682331221 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f82c80453e0 con 0x7f82b407dd00 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.807+0000 7f82c67fc640 1 -- 172.21.15.28:0/682331221 <== osd.6 v2:172.21.15.53:6809/3653668189 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f82c80453e0 con 0x7f82b407dd00 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.808+0000 7f82b3fff640 1 -- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f82b407dd00 msgr2=0x7f82b4080190 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.808+0000 7f82b3fff640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f82b407dd00 0x7f82b4080190 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.808+0000 7f82b3fff640 1 -- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82b40745b0 msgr2=0x7f82b4076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.808+0000 7f82b3fff640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82b40745b0 0x7f82b4076ac0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f82cc005030 tx=0x7f82cc007cc0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.808+0000 7f82b3fff640 1 -- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82c8170830 msgr2=0x7f82c8170c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.808+0000 7f82b3fff640 1 --2- 172.21.15.28:0/682331221 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82c8170830 0x7f82c8170c40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82c8051630 tx=0x7f82c0001970 comp rx=0 tx=0).stop 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.808+0000 7f82d64dc640 1 -- 172.21.15.28:0/682331221 reap_dead start 2024-04-25T19:11:54.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.809+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0ef80b1a50 con 0x7f0edc07dd50 2024-04-25T19:11:54.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.809+0000 7f82b3fff640 1 -- 172.21.15.28:0/682331221 shutdown_connections 2024-04-25T19:11:54.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.809+0000 7f82b3fff640 1 -- 172.21.15.28:0/682331221 >> 172.21.15.28:0/682331221 conn(0x7f82c8054cf0 msgr2=0x7f82c8038300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.809+0000 7f82b3fff640 1 -- 172.21.15.28:0/682331221 shutdown_connections 2024-04-25T19:11:54.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.809+0000 7f82b3fff640 1 -- 172.21.15.28:0/682331221 wait complete. 2024-04-25T19:11:54.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.809+0000 7f22fbd57640 1 -- 172.21.15.28:0/2807250931 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f22dc002dd0 con 0x7f22d807dca0 2024-04-25T19:11:54.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.809+0000 7f0eee7fc640 1 -- 172.21.15.28:0/3964247260 <== osd.5 v2:172.21.15.53:6800/2342659583 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0ef80b1a50 con 0x7f0edc07dd50 2024-04-25T19:11:54.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.810+0000 7f22ea7fc640 1 -- 172.21.15.28:0/2807250931 <== osd.1 v2:172.21.15.28:6808/1898594658 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f22dc002dd0 con 0x7f22d807dca0 2024-04-25T19:11:54.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.810+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f0edc07dd50 msgr2=0x7f0edc0801e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.810+0000 7f0f00185640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f0edc07dd50 0x7f0edc0801e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22d7fff640 1 -- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f22d807dca0 msgr2=0x7f22d8080130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0edc074580 msgr2=0x7f0edc076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f0f00185640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0edc074580 0x7f0edc076a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0ef40011e0 tx=0x7f0ef4009040 comp rx=0 tx=0).stop 2024-04-25T19:11:54.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8164840 msgr2=0x7f0ef814d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f0f00185640 1 --2- 172.21.15.28:0/3964247260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ef8164840 0x7f0ef814d4c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009eb0 tx=0x7f0ef0004a40 comp rx=0 tx=0).stop 2024-04-25T19:11:54.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22d7fff640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f22d807dca0 0x7f22d8080130 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22d7fff640 1 -- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22d80744e0 msgr2=0x7f22d80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22d7fff640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f22d80744e0 0x7f22d80769f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f22f0034000 tx=0x7f22f002e000 comp rx=0 tx=0).stop 2024-04-25T19:11:54.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5cd0002dd0 con 0x7f5cdc07e900 2024-04-25T19:11:54.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22d7fff640 1 -- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22f40cf9e0 msgr2=0x7f22f40a0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22d7fff640 1 --2- 172.21.15.28:0/2807250931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22f40cf9e0 0x7f22f40a0050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00cbc0 tx=0x7f22ec00a430 comp rx=0 tx=0).stop 2024-04-25T19:11:54.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22fa2f4640 1 -- 172.21.15.28:0/2807250931 reap_dead start 2024-04-25T19:11:54.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22d7fff640 1 -- 172.21.15.28:0/2807250931 shutdown_connections 2024-04-25T19:11:54.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.811+0000 7f22d7fff640 1 -- 172.21.15.28:0/2807250931 >> 172.21.15.28:0/2807250931 conn(0x7f22f40b7620 msgr2=0x7f22f4070760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.812+0000 7f0efe722640 1 -- 172.21.15.28:0/3964247260 reap_dead start 2024-04-25T19:11:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.812+0000 7f22d7fff640 1 -- 172.21.15.28:0/2807250931 shutdown_connections 2024-04-25T19:11:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.812+0000 7f22d7fff640 1 -- 172.21.15.28:0/2807250931 wait complete. 2024-04-25T19:11:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.812+0000 7f5cd77fe640 1 -- 172.21.15.28:0/3636179041 <== osd.7 v2:172.21.15.53:6808/79451145 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5cd0002dd0 con 0x7f5cdc07e900 2024-04-25T19:11:54.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.812+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 shutdown_connections 2024-04-25T19:11:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.812+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 >> 172.21.15.28:0/3964247260 conn(0x7f0ef80739e0 msgr2=0x7f0ef806f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.812+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 shutdown_connections 2024-04-25T19:11:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.812+0000 7f0f00185640 1 -- 172.21.15.28:0/3964247260 wait complete. 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.813+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f5cdc07e900 msgr2=0x7f5cdc080df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.813+0000 7f5cf5527640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f5cdc07e900 0x7f5cdc080df0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.813+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cdc0746b0 msgr2=0x7f5cdc076bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.813+0000 7f5cf5527640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cdc0746b0 0x7f5cdc076bc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5cf015e5a0 tx=0x7f5ce4009190 comp rx=0 tx=0).stop 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.814+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 msgr2=0x7f5cf0154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.814+0000 7f5cf5527640 1 --2- 172.21.15.28:0/3636179041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cf006cab0 0x7f5cf0154170 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c820 tx=0x7f5ce000cce0 comp rx=0 tx=0).stop 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.814+0000 7f5cef7fe640 1 -- 172.21.15.28:0/3636179041 reap_dead start 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.814+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 shutdown_connections 2024-04-25T19:11:54.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.814+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 >> 172.21.15.28:0/3636179041 conn(0x7f5cf00739e0 msgr2=0x7f5cf0070980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:54.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.815+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 shutdown_connections 2024-04-25T19:11:54.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:54.815+0000 7f5cf5527640 1 -- 172.21.15.28:0/3636179041 wait complete. 2024-04-25T19:11:54.880 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:54.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T19:11:54.882 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:54.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T19:11:54.883 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:54.883 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:54.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T19:11:54.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T19:11:54.885 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-04-25T19:11:54.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T19:11:54.886 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:54.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T19:11:54.887 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-04-25T19:11:54.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T19:11:54.888 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:54.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T19:11:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.137+0000 7fc2349ef640 1 Processor -- start 2024-04-25T19:11:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.137+0000 7fc2349ef640 1 -- start start 2024-04-25T19:11:55.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.138+0000 7fb221a15640 1 Processor -- start 2024-04-25T19:11:55.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.138+0000 7fb221a15640 1 -- start start 2024-04-25T19:11:55.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.139+0000 7fc2349ef640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2300b0dc0 0x7fc230171040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.139+0000 7fc2349ef640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc2300ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.139+0000 7fc2349ef640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2300aed00 con 0x7fc2300ae8f0 2024-04-25T19:11:55.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.139+0000 7fc2349ef640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fc2300b57f0 con 0x7fc2300b0dc0 2024-04-25T19:11:55.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.139+0000 7fc2349ef640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc2300b5c80 con 0x7fc2300cd880 2024-04-25T19:11:55.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fb221a15640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c06dc90 0x7fb21c0708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fb221a15640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb21c15d0d0 0x7fb21c0b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fb221a15640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb21c0aeb20 0x7fb21c0aef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fb221a15640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb21c0b5660 con 0x7fb21c0aeb20 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fb221a15640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fb21c0af470 con 0x7fb21c15d0d0 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fb221a15640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb21c0b12a0 con 0x7fb21c06dc90 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21b7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb21c0aeb20 0x7fb21c0aef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fc22effd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc2300ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fc22effd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc2300ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47092/0 (socket says 172.21.15.28:47092) 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fc22effd640 1 -- 172.21.15.28:0/920400409 learned_addr learned my addr 172.21.15.28:0/920400409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fc22f7fe640 1 --2- 172.21.15.28:0/920400409 >> v2:172.21.15.53:3300/0 conn(0x7fc2300b0dc0 0x7fc230171040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fc22effd640 1 -- 172.21.15.28:0/920400409 >> v2:172.21.15.53:3300/0 conn(0x7fc2300b0dc0 msgr2=0x7fc230171040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fc22effd640 1 --2- 172.21.15.28:0/920400409 >> v2:172.21.15.53:3300/0 conn(0x7fc2300b0dc0 0x7fc230171040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fc22effd640 1 -- 172.21.15.28:0/920400409 >> v1:172.21.15.53:6789/0 conn(0x7fc2300ae8f0 legacy=0x7fc230174ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T19:11:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.140+0000 7fc22effd640 1 -- 172.21.15.28:0/920400409 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2300b4be0 con 0x7fc2300cd880 2024-04-25T19:11:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21b7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb21c0aeb20 0x7fb21c0aef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37266/0 (socket says 172.21.15.28:37266) 2024-04-25T19:11:55.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21b7fe640 1 -- 172.21.15.28:0/1878125759 learned_addr learned my addr 172.21.15.28:0/1878125759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21a7fc640 1 --2- 172.21.15.28:0/1878125759 >> v2:172.21.15.53:3300/0 conn(0x7fb21c15d0d0 0x7fb21c0b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21affd640 1 --2- 172.21.15.28:0/1878125759 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c06dc90 0x7fb21c0708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21b7fe640 1 -- 172.21.15.28:0/1878125759 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c06dc90 msgr2=0x7fb21c0708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21b7fe640 1 --2- 172.21.15.28:0/1878125759 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c06dc90 0x7fb21c0708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21b7fe640 1 -- 172.21.15.28:0/1878125759 >> v2:172.21.15.53:3300/0 conn(0x7fb21c15d0d0 msgr2=0x7fb21c0b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21b7fe640 1 --2- 172.21.15.28:0/1878125759 >> v2:172.21.15.53:3300/0 conn(0x7fb21c15d0d0 0x7fb21c0b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.141+0000 7fb21b7fe640 1 -- 172.21.15.28:0/1878125759 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21c0b5010 con 0x7fb21c0aeb20 2024-04-25T19:11:55.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.142+0000 7fc22effd640 1 --2- 172.21.15.28:0/920400409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc2300ae3b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc220009870 tx=0x7fc22002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ef52e8469afae65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.142+0000 7fbbc9696640 1 Processor -- start 2024-04-25T19:11:55.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.142+0000 7fc22e7fc640 1 -- 172.21.15.28:0/920400409 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220037070 con 0x7fc2300cd880 2024-04-25T19:11:55.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.142+0000 7fb21b7fe640 1 --2- 172.21.15.28:0/1878125759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c0aeb20 0x7fb21c0aef30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb21c069ed0 tx=0x7fb21002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d116ecbcabc84af3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.142+0000 7fbbc9696640 1 -- start start 2024-04-25T19:11:55.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.143+0000 7fb219ffb640 1 -- 172.21.15.28:0/1878125759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210037070 con 0x7fb21c0aeb20 2024-04-25T19:11:55.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.143+0000 7fb219ffb640 1 -- 172.21.15.28:0/1878125759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb210032440 con 0x7fb21c0aeb20 2024-04-25T19:11:55.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.143+0000 7f90c7d3f640 1 Processor -- start 2024-04-25T19:11:55.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7fa13f59e640 1 Processor -- start 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7fa13f59e640 1 -- start start 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7fb219ffb640 1 -- 172.21.15.28:0/1878125759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb210032810 con 0x7fb21c0aeb20 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7fb221a15640 1 -- 172.21.15.28:0/1878125759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c0aeb20 msgr2=0x7fb21c0aef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7fb221a15640 1 --2- 172.21.15.28:0/1878125759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c0aeb20 0x7fb21c0aef30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fb21c069ed0 tx=0x7fb21002a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7fb221a15640 1 -- 172.21.15.28:0/1878125759 shutdown_connections 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fb221a15640 1 --2- 172.21.15.28:0/1878125759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c0aeb20 0x7fb21c0aef30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fb221a15640 1 --2- 172.21.15.28:0/1878125759 >> v2:172.21.15.53:3300/0 conn(0x7fb21c15d0d0 0x7fb21c0b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fb221a15640 1 --2- 172.21.15.28:0/1878125759 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c06dc90 0x7fb21c0708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fb221a15640 1 -- 172.21.15.28:0/1878125759 >> 172.21.15.28:0/1878125759 conn(0x7fb21c074310 msgr2=0x7fb21c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fb221a15640 1 -- 172.21.15.28:0/1878125759 shutdown_connections 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fb221a15640 1 -- 172.21.15.28:0/1878125759 wait complete. 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.142+0000 7fc22e7fc640 1 -- 172.21.15.28:0/920400409 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc220032440 con 0x7fc2300cd880 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.142+0000 7fc22e7fc640 1 -- 172.21.15.28:0/920400409 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc220032810 con 0x7fc2300cd880 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.143+0000 7fc2349ef640 1 -- 172.21.15.28:0/920400409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 msgr2=0x7fc2300ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.143+0000 7fc2349ef640 1 --2- 172.21.15.28:0/920400409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc2300ae3b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc220009870 tx=0x7fc22002a210 comp rx=0 tx=0).stop 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fc2349ef640 1 -- 172.21.15.28:0/920400409 shutdown_connections 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fc2349ef640 1 --2- 172.21.15.28:0/920400409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc2300ae3b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fc2349ef640 1 --2- 172.21.15.28:0/920400409 >> v2:172.21.15.53:3300/0 conn(0x7fc2300b0dc0 0x7fc230171040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fc2349ef640 1 -- 172.21.15.28:0/920400409 >> 172.21.15.28:0/920400409 conn(0x7fc2300740f0 msgr2=0x7fc23006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7f59e54fc640 1 Processor -- start 2024-04-25T19:11:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7f59e54fc640 1 -- start start 2024-04-25T19:11:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fbbc2ffd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbbc40b6240 0x7fbbc40ae880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40252/0 (socket says 172.21.15.28:40252) 2024-04-25T19:11:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fbbc27fc640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbbc40aef90 0x7fbbc40b4bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58952/0 (socket says 172.21.15.28:58952) 2024-04-25T19:11:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fbbc9696640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbbc40b6920 0x7fbbc415eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fbbc9696640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbc4164e00 con 0x7fbbc40aef90 2024-04-25T19:11:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.145+0000 7fbbc2ffd640 1 -- 172.21.15.28:0/4169822712 learned_addr learned my addr 172.21.15.28:0/4169822712 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:55.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.144+0000 7f90c7d3f640 1 -- start start 2024-04-25T19:11:55.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc9696640 1 -- 172.21.15.28:0/4169822712 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbc41650e0 con 0x7fbbc40b6240 2024-04-25T19:11:55.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc9696640 1 -- 172.21.15.28:0/4169822712 --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fbbc41653c0 con 0x7fbbc40b6920 2024-04-25T19:11:55.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 618969599 0 0) 0x7fbbc4164e00 con 0x7fbbc40aef90 2024-04-25T19:11:55.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7f59e54fc640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f59d8037bb0 0x7f59d8033f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbac003520 con 0x7fbbc40aef90 2024-04-25T19:11:55.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2757065054 0 0) 0x7fbbac003520 con 0x7fbbc40aef90 2024-04-25T19:11:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 >> v2:172.21.15.53:3300/0 conn(0x7fbbc40b6920 msgr2=0x7fbbc415eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc37fe640 1 --2- 172.21.15.28:0/4169822712 >> v2:172.21.15.53:3300/0 conn(0x7fbbc40b6920 0x7fbbc415eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc1ffb640 1 --2- 172.21.15.28:0/4169822712 >> v2:172.21.15.53:3300/0 conn(0x7fbbc40b6920 0x7fbbc415eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 >> v1:172.21.15.53:6789/0 conn(0x7fbbc40b6240 legacy=0x7fbbc40ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc41656a0 con 0x7fbbc40aef90 2024-04-25T19:11:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fc2349ef640 1 -- 172.21.15.28:0/920400409 shutdown_connections 2024-04-25T19:11:55.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fc2349ef640 1 -- 172.21.15.28:0/920400409 wait complete. 2024-04-25T19:11:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f59df7fe640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f59d80060e0 0x7f59d804f340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40260/0 (socket says 172.21.15.28:40260) 2024-04-25T19:11:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f59dffff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f59d8037bb0 0x7f59d8033f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f59dffff640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f59d8037bb0 0x7f59d8033f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53416/0 (socket says 172.21.15.28:53416) 2024-04-25T19:11:55.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13f59e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400ce070 0x7fa1400af030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.146+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fbbbc0019f0 con 0x7fbbc40aef90 2024-04-25T19:11:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 >> v1:172.21.15.28:6789/0 conn(0x7fbbc40aef90 legacy=0x7fbbc40b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fbbc1ffb640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbac003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fbbc1ffb640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbac004800 0x7fbbac004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f90c7d3f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b8034180 0x7f90b8038340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f90c52da640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f90b804f340 0x7f90b8033a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40266/0 (socket says 172.21.15.28:40266) 2024-04-25T19:11:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f90c5adb640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f90b8052fb0 0x7f90b8007320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58968/0 (socket says 172.21.15.28:58968) 2024-04-25T19:11:55.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f90c52da640 1 -- 172.21.15.28:0/874593537 learned_addr learned my addr 172.21.15.28:0/874593537 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13f59e640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa1400af570 0x7fa1400d1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13f59e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa14006d480 con 0x7fa1400b2460 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13f59e640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fa1400b2020 con 0x7fa1400af570 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13f59e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa14006e150 con 0x7fa1400ce070 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13dd9b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400ce070 0x7fa1400af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13dd9b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400ce070 0x7fa1400af030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47126/0 (socket says 172.21.15.28:47126) 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13dd9b640 1 -- 172.21.15.28:0/3385001268 learned_addr learned my addr 172.21.15.28:0/3385001268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fa13ed9d640 1 --2- 172.21.15.28:0/3385001268 >> v2:172.21.15.53:3300/0 conn(0x7fa1400af570 0x7fa1400d1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f59df7fe640 1 -- 172.21.15.28:0/2056310216 learned_addr learned my addr 172.21.15.28:0/2056310216 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f59e54fc640 1 -- 172.21.15.28:0/2056310216 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d8050db0 con 0x7f59d804fa50 2024-04-25T19:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f59e54fc640 1 -- 172.21.15.28:0/2056310216 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d8051090 con 0x7f59d80060e0 2024-04-25T19:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f59e54fc640 1 -- 172.21.15.28:0/2056310216 --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f59d803a460 con 0x7f59d8037bb0 2024-04-25T19:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f59deffd640 1 -- 172.21.15.28:0/2056310216 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2610652610 0 0) 0x7f59d8051090 con 0x7f59d80060e0 2024-04-25T19:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7f90c62dc640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b8034180 0x7f90b8038340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fbbc1ffb640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbac0099b0 0x7fbbac00bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc41653c0 con 0x7fbbac003a00 2024-04-25T19:11:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbbac00c3f0 con 0x7fbbac0099b0 2024-04-25T19:11:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbbac00c5f0 con 0x7fbbac004800 2024-04-25T19:11:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f90c7d3f640 1 -- 172.21.15.28:0/874593537 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b804f750 con 0x7f90b8052fb0 2024-04-25T19:11:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.147+0000 7fbbc37fe640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbac0099b0 0x7fbbac00bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f90c7d3f640 1 -- 172.21.15.28:0/874593537 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90b8007a30 con 0x7f90b804f340 2024-04-25T19:11:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f90c7d3f640 1 -- 172.21.15.28:0/874593537 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f90b80533c0 con 0x7f90b8034180 2024-04-25T19:11:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7fb221a15640 1 Processor -- start 2024-04-25T19:11:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7fb221a15640 1 -- start start 2024-04-25T19:11:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7fbbc2ffd640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbac003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7fbbc27fc640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbac004800 0x7fbbac004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1767244204 0 0) 0x7f90b8007a30 con 0x7f90b804f340 2024-04-25T19:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90ac003520 con 0x7f90b804f340 2024-04-25T19:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3373540107 0 0) 0x7f90b804f750 con 0x7f90b8052fb0 2024-04-25T19:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90b8007a30 con 0x7f90b8052fb0 2024-04-25T19:11:55.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f59dffff640 1 -- 172.21.15.28:0/2056310216 >> v1:172.21.15.53:6789/0 conn(0x7f59d80060e0 legacy=0x7f59d804f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7f59dffff640 1 -- 172.21.15.28:0/2056310216 >> v1:172.21.15.28:6789/0 conn(0x7f59d804fa50 legacy=0x7f59d80528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f59dffff640 1 -- 172.21.15.28:0/2056310216 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d8051370 con 0x7f59d8037bb0 2024-04-25T19:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7fbbc2ffd640 1 -- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbac004800 msgr2=0x7fbbac004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7fbbc2ffd640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbac004800 0x7fbbac004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7fbbc2ffd640 1 -- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbac0099b0 msgr2=0x7fbbac00bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7fbbc2ffd640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbac0099b0 0x7fbbac00bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7fbbc2ffd640 1 -- 172.21.15.28:0/4169822712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbac003520 con 0x7fbbac003a00 2024-04-25T19:11:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2001180117 0 0) 0x7f90b8007a30 con 0x7f90b8052fb0 2024-04-25T19:11:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b8034180 msgr2=0x7f90b8038340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b8034180 0x7f90b8038340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 >> v1:172.21.15.53:6789/0 conn(0x7f90b804f340 legacy=0x7f90b8033a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b80e8230 con 0x7f90b8052fb0 2024-04-25T19:11:55.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f90bc0015f0 con 0x7f90b8052fb0 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 >> v1:172.21.15.28:6789/0 conn(0x7f90b8052fb0 legacy=0x7f90b8007320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90ac004c00 0x7f90ac005030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac0037a0 0x7f90ac003bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ac004340 0x7f90ac00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90b80533c0 con 0x7f90ac004340 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f90ac00c340 con 0x7f90ac004c00 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f90ac00c540 con 0x7f90ac0037a0 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c5adb640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90ac004c00 0x7f90ac005030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f90c52da640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac0037a0 0x7f90ac003bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7fa13ed9d640 1 -- 172.21.15.28:0/3385001268 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400ce070 msgr2=0x7fa1400af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7fa13ed9d640 1 --2- 172.21.15.28:0/3385001268 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400ce070 0x7fa1400af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7fa13ed9d640 1 -- 172.21.15.28:0/3385001268 >> v1:172.21.15.28:6789/0 conn(0x7fa1400b2460 legacy=0x7fa14006da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.148+0000 7fa13ed9d640 1 -- 172.21.15.28:0/3385001268 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14009b790 con 0x7fa1400af570 2024-04-25T19:11:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f41b3fff640 1 Processor -- start 2024-04-25T19:11:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7f41b3fff640 1 -- start start 2024-04-25T19:11:55.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.149+0000 7fa13ed9d640 1 --2- 172.21.15.28:0/3385001268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1400d1dc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa138009d90 tx=0x7fa13802a590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2382c77cafe30736 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7fbbc37fe640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbac0099b0 0x7fbbac00bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7fbbc37fe640 1 -- 172.21.15.28:0/4169822712 reap_dead start 2024-04-25T19:11:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f917035e640 1 Processor -- start 2024-04-25T19:11:55.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f917035e640 1 -- start start 2024-04-25T19:11:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7fc2349ef640 1 Processor -- start 2024-04-25T19:11:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f90c5adb640 1 -- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac0037a0 msgr2=0x7f90ac003bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f90c5adb640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac0037a0 0x7f90ac003bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f90c5adb640 1 -- 172.21.15.28:0/874593537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ac004340 msgr2=0x7f90ac00ba60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T19:11:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f90c5adb640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90ac004340 0x7f90ac00ba60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f90c5adb640 1 -- 172.21.15.28:0/874593537 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b8007a30 con 0x7f90ac004c00 2024-04-25T19:11:55.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f90c62dc640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90b8034180 0x7f90b8038340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7fc2349ef640 1 -- start start 2024-04-25T19:11:55.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7fbbc2ffd640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbac003e70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4071850 tx=0x7fbbac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f62a9cd6f6b9c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f59dffff640 1 --2- 172.21.15.28:0/2056310216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 0x7f59d8033f10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59d40097a0 tx=0x7f59d4029f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76cbb89e165df9c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7f59deffd640 1 -- 172.21.15.28:0/2056310216 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4037070 con 0x7f59d8037bb0 2024-04-25T19:11:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7f59deffd640 1 -- 172.21.15.28:0/2056310216 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d402abf0 con 0x7f59d8037bb0 2024-04-25T19:11:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7f59deffd640 1 -- 172.21.15.28:0/2056310216 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4037070 con 0x7f59d8037bb0 2024-04-25T19:11:55.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fb221a15640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c06dc90 0x7fb21c153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fa13d59a640 1 -- 172.21.15.28:0/3385001268 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138007740 con 0x7fa1400af570 2024-04-25T19:11:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fa13d59a640 1 -- 172.21.15.28:0/3385001268 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1380078e0 con 0x7fa1400af570 2024-04-25T19:11:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fa13d59a640 1 -- 172.21.15.28:0/3385001268 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138007cf0 con 0x7fa1400af570 2024-04-25T19:11:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fb21affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c06dc90 0x7fb21c153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fb21affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c06dc90 0x7fb21c153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37274/0 (socket says 172.21.15.28:37274) 2024-04-25T19:11:55.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fb21affd640 1 -- 172.21.15.28:0/1494504966 learned_addr learned my addr 172.21.15.28:0/1494504966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fb221a15640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c0aeb20 0x7fb21c14f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fb221a15640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb21c15d0d0 0x7fb21c09dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.150+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4001460 con 0x7fbbac003a00 2024-04-25T19:11:55.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb402e070 con 0x7fbbac003a00 2024-04-25T19:11:55.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.151+0000 7fbbc1ffb640 1 -- 172.21.15.28:0/4169822712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb402ad70 con 0x7fbbac003a00 2024-04-25T19:11:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c0b6030 con 0x7fb21c06dc90 2024-04-25T19:11:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fb21a7fc640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c0aeb20 0x7fb21c14f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f90c5adb640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90ac004c00 0x7f90ac005030 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90bc007f80 tx=0x7f90ac005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0a7d56912bbc175 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f59e54fc640 1 -- 172.21.15.28:0/2056310216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 msgr2=0x7f59d8033f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f59e54fc640 1 --2- 172.21.15.28:0/2056310216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 0x7f59d8033f10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59d40097a0 tx=0x7f59d4029f60 comp rx=0 tx=0).stop 2024-04-25T19:11:55.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fbbc9696640 1 -- 172.21.15.28:0/4169822712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 msgr2=0x7fbbac003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fbbc9696640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbac003e70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4071850 tx=0x7fbbac005830 comp rx=0 tx=0).stop 2024-04-25T19:11:55.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fb21b7fe640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb21c15d0d0 0x7fb21c09dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc007ba0 con 0x7f90ac004c00 2024-04-25T19:11:55.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f41b3fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41ac0e5180 0x7f41ac0e1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fa13f59e640 1 -- 172.21.15.28:0/3385001268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 msgr2=0x7fa1400d1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fa13f59e640 1 --2- 172.21.15.28:0/3385001268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1400d1dc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa138009d90 tx=0x7fa13802a590 comp rx=0 tx=0).stop 2024-04-25T19:11:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f41b3fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0e2650 0x7f41ac051db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90bc007290 con 0x7f90ac004c00 2024-04-25T19:11:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f90c4ad9640 1 -- 172.21.15.28:0/874593537 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90bc007680 con 0x7f90ac004c00 2024-04-25T19:11:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f59e54fc640 1 -- 172.21.15.28:0/2056310216 shutdown_connections 2024-04-25T19:11:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f59e54fc640 1 --2- 172.21.15.28:0/2056310216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 0x7f59d8033f10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fbbc9696640 1 -- 172.21.15.28:0/4169822712 shutdown_connections 2024-04-25T19:11:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fbbc9696640 1 --2- 172.21.15.28:0/4169822712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbac003e70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fbbc9696640 1 -- 172.21.15.28:0/4169822712 >> 172.21.15.28:0/4169822712 conn(0x7fbbc40742c0 msgr2=0x7fbbc406c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f59e54fc640 1 -- 172.21.15.28:0/2056310216 >> 172.21.15.28:0/2056310216 conn(0x7f59d8054cf0 msgr2=0x7f59d8055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fbbc9696640 1 -- 172.21.15.28:0/4169822712 shutdown_connections 2024-04-25T19:11:55.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fbbc9696640 1 -- 172.21.15.28:0/4169822712 wait complete. 2024-04-25T19:11:55.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fa13f59e640 1 -- 172.21.15.28:0/3385001268 shutdown_connections 2024-04-25T19:11:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f41b37fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f41ac0522f0 0x7f41ac055b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59000/0 (socket says 172.21.15.28:59000) 2024-04-25T19:11:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fa13f59e640 1 --2- 172.21.15.28:0/3385001268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1400d1dc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fa13f59e640 1 --2- 172.21.15.28:0/3385001268 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400ce070 0x7fa1400af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fa13f59e640 1 -- 172.21.15.28:0/3385001268 >> 172.21.15.28:0/3385001268 conn(0x7fa140073b80 msgr2=0x7fa1400b7300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f41b37fe640 1 -- 172.21.15.28:0/2278613270 learned_addr learned my addr 172.21.15.28:0/2278613270 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:55.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7fc2349ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300ae8f0 0x7fc230207440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f917035e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916809b540 0x7f916809b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.152+0000 7f41b3fff640 1 -- 172.21.15.28:0/2278613270 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41ac052700 con 0x7f41ac0522f0 2024-04-25T19:11:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b3fff640 1 -- 172.21.15.28:0/2278613270 --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f41ac0e5590 con 0x7f41ac0e5180 2024-04-25T19:11:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b27fc640 1 --2- 172.21.15.28:0/2278613270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0e2650 0x7f41ac051db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b3fff640 1 -- 172.21.15.28:0/2278613270 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f41ac006210 con 0x7f41ac0e2650 2024-04-25T19:11:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f916e0fa640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f91680cfa00 0x7f91680b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59004/0 (socket says 172.21.15.28:59004) 2024-04-25T19:11:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f916d8f9640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f916815e220 0x7f9168157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40278/0 (socket says 172.21.15.28:40278) 2024-04-25T19:11:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fa13f59e640 1 -- 172.21.15.28:0/3385001268 shutdown_connections 2024-04-25T19:11:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b2ffd640 1 --2- 172.21.15.28:0/2278613270 >> v2:172.21.15.28:3300/0 conn(0x7f41ac0e5180 0x7f41ac0e1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f917035e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f916815e630 con 0x7f91680cfa00 2024-04-25T19:11:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fa13f59e640 1 -- 172.21.15.28:0/3385001268 wait complete. 2024-04-25T19:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c0b6720 con 0x7fb21c15d0d0 2024-04-25T19:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb21c150f40 con 0x7fb21c0aeb20 2024-04-25T19:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f916e8fb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916809b540 0x7f916809b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f916e8fb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916809b540 0x7f916809b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47166/0 (socket says 172.21.15.28:47166) 2024-04-25T19:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f916e8fb640 1 -- 172.21.15.28:0/2216915189 learned_addr learned my addr 172.21.15.28:0/2216915189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f917035e640 1 -- 172.21.15.28:0/2216915189 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f916809be90 con 0x7f916815e220 2024-04-25T19:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f917035e640 1 -- 172.21.15.28:0/2216915189 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f91680cffe0 con 0x7f916809b540 2024-04-25T19:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b27fc640 1 -- 172.21.15.28:0/2278613270 >> v2:172.21.15.28:3300/0 conn(0x7f41ac0e5180 msgr2=0x7f41ac0e1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b27fc640 1 --2- 172.21.15.28:0/2278613270 >> v2:172.21.15.28:3300/0 conn(0x7f41ac0e5180 0x7f41ac0e1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b27fc640 1 -- 172.21.15.28:0/2278613270 >> v1:172.21.15.28:6789/0 conn(0x7f41ac0522f0 legacy=0x7f41ac055b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b27fc640 1 -- 172.21.15.28:0/2278613270 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac0e57d0 con 0x7f41ac0e2650 2024-04-25T19:11:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f59e54fc640 1 -- 172.21.15.28:0/2056310216 shutdown_connections 2024-04-25T19:11:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f41b2ffd640 1 --2- 172.21.15.28:0/2278613270 >> v2:172.21.15.28:3300/0 conn(0x7f41ac0e5180 0x7f41ac0e1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f916e8fb640 1 -- 172.21.15.28:0/2216915189 >> v1:172.21.15.53:6789/0 conn(0x7f916815e220 legacy=0x7f9168157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f916e8fb640 1 -- 172.21.15.28:0/2216915189 >> v1:172.21.15.28:6789/0 conn(0x7f91680cfa00 legacy=0x7f91680b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f90c7d3f640 1 -- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90ac004c00 msgr2=0x7f90ac005030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f90c7d3f640 1 --2- 172.21.15.28:0/874593537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90ac004c00 0x7f90ac005030 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f90bc007f80 tx=0x7f90ac005830 comp rx=0 tx=0).stop 2024-04-25T19:11:55.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc2349ef640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2300b0dc0 0x7fc230166420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc2349ef640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc23008b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc22effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2300b0dc0 0x7fc230166420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc22effd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2300b0dc0 0x7fc230166420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53462/0 (socket says 172.21.15.28:53462) 2024-04-25T19:11:55.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc22effd640 1 -- 172.21.15.28:0/122950807 learned_addr learned my addr 172.21.15.28:0/122950807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc2349ef640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc2300b5480 con 0x7fc2300ae8f0 2024-04-25T19:11:55.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f90c62dc640 1 -- 172.21.15.28:0/874593537 reap_dead start 2024-04-25T19:11:55.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc2349ef640 1 -- 172.21.15.28:0/122950807 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc2300b51a0 con 0x7fc2300b0dc0 2024-04-25T19:11:55.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc2349ef640 1 -- 172.21.15.28:0/122950807 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc23006feb0 con 0x7fc2300cd880 2024-04-25T19:11:55.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7fc22f7fe640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300ae8f0 0x7fc230207440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7f59e54fc640 1 -- 172.21.15.28:0/2056310216 wait complete. 2024-04-25T19:11:55.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fb21a7fc640 1 -- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb21c15d0d0 msgr2=0x7fb21c09dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fb21a7fc640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb21c15d0d0 0x7fb21c09dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fb21a7fc640 1 -- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c06dc90 msgr2=0x7fb21c153a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fb21a7fc640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c06dc90 0x7fb21c153a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fb21a7fc640 1 -- 172.21.15.28:0/1494504966 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21c1543b0 con 0x7fb21c0aeb20 2024-04-25T19:11:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fc22ffff640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc23008b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fb21a7fc640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c0aeb20 0x7fb21c14f410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb21400eba0 tx=0x7fb21400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fc22ffff640 1 -- 172.21.15.28:0/122950807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2300b0dc0 msgr2=0x7fc230166420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7f90c7d3f640 1 -- 172.21.15.28:0/874593537 shutdown_connections 2024-04-25T19:11:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7f90c7d3f640 1 -- 172.21.15.28:0/874593537 >> 172.21.15.28:0/874593537 conn(0x7f90b8054cf0 msgr2=0x7f90b8055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fc22ffff640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2300b0dc0 0x7fc230166420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fb203fff640 1 -- 172.21.15.28:0/1494504966 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214018020 con 0x7fb21c0aeb20 2024-04-25T19:11:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7f41b27fc640 1 --2- 172.21.15.28:0/2278613270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0e2650 0x7f41ac051db0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f41a80097a0 tx=0x7f41a802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7a14cb57ee4ccb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fc22ffff640 1 -- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300ae8f0 msgr2=0x7fc230207440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fc22ffff640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300ae8f0 0x7fc230207440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7f90c7d3f640 1 -- 172.21.15.28:0/874593537 shutdown_connections 2024-04-25T19:11:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.153+0000 7f916e8fb640 1 -- 172.21.15.28:0/2216915189 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91680d02c0 con 0x7f916809b540 2024-04-25T19:11:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7fb203fff640 1 -- 172.21.15.28:0/1494504966 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb21400cda0 con 0x7fb21c0aeb20 2024-04-25T19:11:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb21c154690 con 0x7fb21c0aeb20 2024-04-25T19:11:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb21c154c50 con 0x7fb21c0aeb20 2024-04-25T19:11:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fc22ffff640 1 -- 172.21.15.28:0/122950807 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc230166bc0 con 0x7fc2300cd880 2024-04-25T19:11:55.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7f916e8fb640 1 --2- 172.21.15.28:0/2216915189 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916809b540 0x7f916809b950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9160009e80 tx=0x7f916002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cd11e7c770d36c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7fb203fff640 1 -- 172.21.15.28:0/1494504966 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2140146c0 con 0x7fb21c0aeb20 2024-04-25T19:11:55.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.154+0000 7fc22f7fe640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300ae8f0 0x7fc230207440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:55.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7fc22ffff640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc23008b4a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc22400b3e0 tx=0x7fc22400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7f90c7d3f640 1 -- 172.21.15.28:0/874593537 wait complete. 2024-04-25T19:11:55.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7f916d0f8640 1 -- 172.21.15.28:0/2216915189 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160031600 con 0x7f916809b540 2024-04-25T19:11:55.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7f916d0f8640 1 -- 172.21.15.28:0/2216915189 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91600317a0 con 0x7f916809b540 2024-04-25T19:11:55.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.156+0000 7f916d0f8640 1 -- 172.21.15.28:0/2216915189 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160031b70 con 0x7f916809b540 2024-04-25T19:11:55.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.156+0000 7fbbc9696640 1 Processor -- start 2024-04-25T19:11:55.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.156+0000 7fbbc9696640 1 -- start start 2024-04-25T19:11:55.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.156+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb21c154120 con 0x7fb21c0aeb20 2024-04-25T19:11:55.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.155+0000 7f41b1ffb640 1 -- 172.21.15.28:0/2278613270 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8037070 con 0x7f41ac0e2650 2024-04-25T19:11:55.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.156+0000 7f41b1ffb640 1 -- 172.21.15.28:0/2278613270 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a8032440 con 0x7f41ac0e2650 2024-04-25T19:11:55.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.156+0000 7f41b1ffb640 1 -- 172.21.15.28:0/2278613270 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8032810 con 0x7f41ac0e2650 2024-04-25T19:11:55.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7fa13f59e640 1 Processor -- start 2024-04-25T19:11:55.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f917035e640 1 -- 172.21.15.28:0/2216915189 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916809b540 msgr2=0x7f916809b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f917035e640 1 --2- 172.21.15.28:0/2216915189 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916809b540 0x7f916809b950 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9160009e80 tx=0x7f916002a530 comp rx=0 tx=0).stop 2024-04-25T19:11:55.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f59e54fc640 1 Processor -- start 2024-04-25T19:11:55.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7fa13f59e640 1 -- start start 2024-04-25T19:11:55.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f41b3fff640 1 -- 172.21.15.28:0/2278613270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0e2650 msgr2=0x7f41ac051db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f41b3fff640 1 --2- 172.21.15.28:0/2278613270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0e2650 0x7f41ac051db0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f41a80097a0 tx=0x7f41a802a1a0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f917035e640 1 -- 172.21.15.28:0/2216915189 shutdown_connections 2024-04-25T19:11:55.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f917035e640 1 --2- 172.21.15.28:0/2216915189 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916809b540 0x7f916809b950 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f59e54fc640 1 -- start start 2024-04-25T19:11:55.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f917035e640 1 -- 172.21.15.28:0/2216915189 >> 172.21.15.28:0/2216915189 conn(0x7f91680b7620 msgr2=0x7f91680b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7fc217fff640 1 -- 172.21.15.28:0/122950807 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224015020 con 0x7fc2300cd880 2024-04-25T19:11:55.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7fc217fff640 1 -- 172.21.15.28:0/122950807 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc224011850 con 0x7fc2300cd880 2024-04-25T19:11:55.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7fc217fff640 1 -- 172.21.15.28:0/122950807 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224011c40 con 0x7fc2300cd880 2024-04-25T19:11:55.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7fc2349ef640 1 -- 172.21.15.28:0/122950807 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc230166ea0 con 0x7fc2300cd880 2024-04-25T19:11:55.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f90c7d3f640 1 Processor -- start 2024-04-25T19:11:55.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7f917035e640 1 -- 172.21.15.28:0/2216915189 shutdown_connections 2024-04-25T19:11:55.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7fb203fff640 1 -- 172.21.15.28:0/1494504966 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb214005d10 con 0x7fb21c0aeb20 2024-04-25T19:11:55.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.157+0000 7fc2349ef640 1 -- 172.21.15.28:0/122950807 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc230167460 con 0x7fc2300cd880 2024-04-25T19:11:55.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7f917035e640 1 -- 172.21.15.28:0/2216915189 wait complete. 2024-04-25T19:11:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7f41b3fff640 1 -- 172.21.15.28:0/2278613270 shutdown_connections 2024-04-25T19:11:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7f41b3fff640 1 --2- 172.21.15.28:0/2278613270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0e2650 0x7f41ac051db0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7f41b3fff640 1 --2- 172.21.15.28:0/2278613270 >> v2:172.21.15.28:3300/0 conn(0x7f41ac0e5180 0x7f41ac0e1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7f41b3fff640 1 -- 172.21.15.28:0/2278613270 >> 172.21.15.28:0/2278613270 conn(0x7f41ac00c3f0 msgr2=0x7f41ac00c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7f90c7d3f640 1 -- start start 2024-04-25T19:11:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7f41b3fff640 1 -- 172.21.15.28:0/2278613270 shutdown_connections 2024-04-25T19:11:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7f41b3fff640 1 -- 172.21.15.28:0/2278613270 wait complete. 2024-04-25T19:11:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fb203fff640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2080745b0 0x7fb208076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fb203fff640 1 -- 172.21.15.28:0/1494504966 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb214096110 con 0x7fb21c0aeb20 2024-04-25T19:11:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fb21affd640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2080745b0 0x7fb208076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc9696640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbc4144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbc4144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbc4144210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37304/0 (socket says 172.21.15.28:37304) 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc2ffd640 1 -- 172.21.15.28:0/3285888950 learned_addr learned my addr 172.21.15.28:0/3285888950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc9696640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc415b840 0x7fbbc41569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc9696640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc415bc50 0x7fbbc4143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc4164ba0 con 0x7fbbac003a00 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc4164a20 con 0x7fbbc415bc50 2024-04-25T19:11:55.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.158+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc415eb00 con 0x7fbbc415b840 2024-04-25T19:11:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.159+0000 7fb21affd640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2080745b0 0x7fb208076ac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb20c007a40 tx=0x7fb20c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.159+0000 7fbbc27fc640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc415b840 0x7fbbc41569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.159+0000 7fbbc37fe640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc415bc50 0x7fbbc4143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.159+0000 7fbbc2ffd640 1 -- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc415b840 msgr2=0x7fbbc41569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.159+0000 7fbbc2ffd640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc415b840 0x7fbbc41569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.159+0000 7fbbc2ffd640 1 -- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc415bc50 msgr2=0x7fbbc4143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.159+0000 7fbbc2ffd640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc415bc50 0x7fbbc4143720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.159+0000 7fbbc2ffd640 1 -- 172.21.15.28:0/3285888950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc415a750 con 0x7fbbac003a00 2024-04-25T19:11:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e54fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59d80060e0 0x7f59d80d9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fa13f59e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1401ee720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e54fc640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 0x7f59d80cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fa13f59e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400b2460 0x7fa1400d1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fbbc2ffd640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbc4144210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40714e0 tx=0x7fbbb4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fa13e59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1401ee720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fa13e59c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1401ee720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53480/0 (socket says 172.21.15.28:53480) 2024-04-25T19:11:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fa13e59c640 1 -- 172.21.15.28:0/2633943184 learned_addr learned my addr 172.21.15.28:0/2633943184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f90c7d3f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac004c00 0x7f90b8171b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fa13dd9b640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400b2460 0x7fa1400d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f90c7d3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90b81809c0 0x7f90b8180dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fa13f59e640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1400ce070 0x7fa140150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f90c7d3f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b80e5240 0x7f90b80e5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f90c7d3f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90b8006840 con 0x7f90b81809c0 2024-04-25T19:11:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f90c7d3f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f90b8006560 con 0x7f90b80e5240 2024-04-25T19:11:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f90c7d3f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f90b8006b20 con 0x7f90ac004c00 2024-04-25T19:11:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e54fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d80c81c0 0x7f59d80c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e54fc640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f59d803a8f0 con 0x7f59d80c81c0 2024-04-25T19:11:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e54fc640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f59d80064f0 con 0x7f59d8037bb0 2024-04-25T19:11:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e54fc640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f59d80d9a40 con 0x7f59d80060e0 2024-04-25T19:11:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7fbbab7fe640 1 -- 172.21.15.28:0/3285888950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4043070 con 0x7fbbac003a00 2024-04-25T19:11:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f90c5adb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac004c00 0x7f90b8171b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e4cfb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d80c81c0 0x7f59d80c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e4cfb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d80c81c0 0x7f59d80c9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37314/0 (socket says 172.21.15.28:37314) 2024-04-25T19:11:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59e4cfb640 1 -- 172.21.15.28:0/1155374226 learned_addr learned my addr 172.21.15.28:0/1155374226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59df7fe640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 0x7f59d80cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.160+0000 7f59dffff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59d80060e0 0x7f59d80d9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c52da640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90b81809c0 0x7f90b8180dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c5adb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac004c00 0x7f90b8171b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47202/0 (socket says 172.21.15.28:47202) 2024-04-25T19:11:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c5adb640 1 -- 172.21.15.28:0/3824554104 learned_addr learned my addr 172.21.15.28:0/3824554104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c62dc640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b80e5240 0x7f90b80e5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7fbbab7fe640 1 -- 172.21.15.28:0/3285888950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb4004ae0 con 0x7fbbac003a00 2024-04-25T19:11:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7fbbab7fe640 1 -- 172.21.15.28:0/3285888950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4007070 con 0x7fbbac003a00 2024-04-25T19:11:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f917035e640 1 Processor -- start 2024-04-25T19:11:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f917035e640 1 -- start start 2024-04-25T19:11:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f59dffff640 1 -- 172.21.15.28:0/1155374226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 msgr2=0x7f59d80cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f59dffff640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 0x7f59d80cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f59dffff640 1 -- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d80c81c0 msgr2=0x7f59d80c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f59dffff640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d80c81c0 0x7f59d80c9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f59dffff640 1 -- 172.21.15.28:0/1155374226 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d80cb730 con 0x7f59d80060e0 2024-04-25T19:11:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7fc2349ef640 1 -- 172.21.15.28:0/122950807 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc230050740 con 0x7fc2300cd880 2024-04-25T19:11:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c52da640 1 -- 172.21.15.28:0/3824554104 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac004c00 msgr2=0x7f90b8171b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c52da640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac004c00 0x7f90b8171b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c52da640 1 -- 172.21.15.28:0/3824554104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b80e5240 msgr2=0x7f90b80e5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c52da640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b80e5240 0x7f90b80e5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7f90c52da640 1 -- 172.21.15.28:0/3824554104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b80e5b90 con 0x7f90b81809c0 2024-04-25T19:11:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc4155630 con 0x7fbbac003a00 2024-04-25T19:11:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc415a190 con 0x7fbbac003a00 2024-04-25T19:11:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7f41b3fff640 1 Processor -- start 2024-04-25T19:11:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7f59dffff640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59d80060e0 0x7f59d80d9180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f59d8007cb0 tx=0x7f59d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7f90c52da640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90b81809c0 0x7f90b8180dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f90b0001270 tx=0x7f90b0007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7f41b3fff640 1 -- start start 2024-04-25T19:11:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7f90b67fc640 1 -- 172.21.15.28:0/3824554104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b000f780 con 0x7f90b81809c0 2024-04-25T19:11:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7fa13f59e640 1 -- 172.21.15.28:0/2633943184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1400af980 con 0x7fa1400ce070 2024-04-25T19:11:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7fa13f59e640 1 -- 172.21.15.28:0/2633943184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa14009b540 con 0x7fa1400af570 2024-04-25T19:11:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.161+0000 7fa13f59e640 1 -- 172.21.15.28:0/2633943184 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa14009ba70 con 0x7fa1400b2460 2024-04-25T19:11:55.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b804daf0 con 0x7f90b81809c0 2024-04-25T19:11:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b816f320 con 0x7f90b81809c0 2024-04-25T19:11:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7fa13ed9d640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1400ce070 0x7fa140150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.162+0000 7fa13dd9b640 1 -- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 msgr2=0x7fa1401ee720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc4050740 con 0x7fbbac003a00 2024-04-25T19:11:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7f917035e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916809b540 0x7f916814bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7f917035e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91680cfa00 0x7f91681525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7f917035e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916815e220 0x7f9168154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7fa13dd9b640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1401ee720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7fa13dd9b640 1 -- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1400ce070 msgr2=0x7fa140150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7fa13dd9b640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1400ce070 0x7fa140150fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7fa13dd9b640 1 -- 172.21.15.28:0/2633943184 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa140151500 con 0x7fa1400b2460 2024-04-25T19:11:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7f917035e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f916815d800 con 0x7f91680cfa00 2024-04-25T19:11:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7f90b67fc640 1 -- 172.21.15.28:0/3824554104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b0018070 con 0x7f90b81809c0 2024-04-25T19:11:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7f90b67fc640 1 -- 172.21.15.28:0/3824554104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b000fb20 con 0x7f90b81809c0 2024-04-25T19:11:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7f917035e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f916815d680 con 0x7f916809b540 2024-04-25T19:11:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f41b3fff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0522f0 0x7f41ac0d8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f917035e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f916806d680 con 0x7f916815e220 2024-04-25T19:11:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.163+0000 7fa13e59c640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1401ee720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f916e0fa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916809b540 0x7f916814bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f916e0fa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916809b540 0x7f916814bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:53502/0 (socket says 172.21.15.28:53502) 2024-04-25T19:11:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f916e0fa640 1 -- 172.21.15.28:0/2426175603 learned_addr learned my addr 172.21.15.28:0/2426175603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f41b3fff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41ac0e2650 0x7f41ac0dd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7fa13ed9d640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1400ce070 0x7fa140150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f41b2ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0522f0 0x7f41ac0d8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f916e8fb640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916815e220 0x7f9168154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f41b2ffd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0522f0 0x7f41ac0d8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47220/0 (socket says 172.21.15.28:47220) 2024-04-25T19:11:55.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f41b2ffd640 1 -- 172.21.15.28:0/1556001583 learned_addr learned my addr 172.21.15.28:0/1556001583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7fb203fff640 1 -- 172.21.15.28:0/1494504966 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb214022030 con 0x7fb21c0aeb20 2024-04-25T19:11:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f916d8f9640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91680cfa00 0x7f91681525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f41b27fc640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41ac0e2650 0x7f41ac0dd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f41b3fff640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac0e5180 0x7f41ac0dd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7fa13dd9b640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400b2460 0x7fa1400d1300 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa13400e8a0 tx=0x7fa13400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f916e8fb640 1 -- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916809b540 msgr2=0x7f916814bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f916e8fb640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916809b540 0x7f916814bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f916e8fb640 1 -- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91680cfa00 msgr2=0x7f91681525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f916e8fb640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91680cfa00 0x7f91681525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f916e8fb640 1 -- 172.21.15.28:0/2426175603 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f916814fa00 con 0x7f916815e220 2024-04-25T19:11:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f916e0fa640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916809b540 0x7f916814bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f916e8fb640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916815e220 0x7f9168154560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9160030000 tx=0x7f9160004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f41b37fe640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac0e5180 0x7f41ac0dd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f41b3fff640 1 -- 172.21.15.28:0/1556001583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac0e6070 con 0x7f41ac0e5180 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f41b3fff640 1 -- 172.21.15.28:0/1556001583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac0e5d90 con 0x7f41ac0e2650 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f41b3fff640 1 -- 172.21.15.28:0/1556001583 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f41ac0e2a60 con 0x7f41ac0522f0 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f41b27fc640 1 -- 172.21.15.28:0/1556001583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0522f0 msgr2=0x7f41ac0d8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f41b27fc640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0522f0 0x7f41ac0d8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f41b27fc640 1 -- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac0e5180 msgr2=0x7f41ac0dd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f41b27fc640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac0e5180 0x7f41ac0dd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f41b27fc640 1 -- 172.21.15.28:0/1556001583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac0de110 con 0x7f41ac0e2650 2024-04-25T19:11:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f41b37fe640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac0e5180 0x7f41ac0dd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7f59dcff9640 1 -- 172.21.15.28:0/1155374226 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4037070 con 0x7f59d80060e0 2024-04-25T19:11:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f59dcff9640 1 -- 172.21.15.28:0/1155374226 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d4004030 con 0x7f59d80060e0 2024-04-25T19:11:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f59dcff9640 1 -- 172.21.15.28:0/1155374226 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4004400 con 0x7f59d80060e0 2024-04-25T19:11:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d80c8750 con 0x7f59d80060e0 2024-04-25T19:11:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.164+0000 7fbbab7fe640 1 -- 172.21.15.28:0/3285888950 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbbb4030070 con 0x7fbbac003a00 2024-04-25T19:11:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7fbbab7fe640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbb0074500 0x7fbbb0076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7fbbc27fc640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbb0074500 0x7fbbb0076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f41b27fc640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41ac0e2650 0x7f41ac0dd390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f41ac006390 tx=0x7f41a80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f419b7fe640 1 -- 172.21.15.28:0/1556001583 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8037050 con 0x7f41ac0e2650 2024-04-25T19:11:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f41b3fff640 1 -- 172.21.15.28:0/1556001583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ac0de3f0 con 0x7f41ac0e2650 2024-04-25T19:11:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7fbbab7fe640 1 -- 172.21.15.28:0/3285888950 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbbb407b220 con 0x7fbbac003a00 2024-04-25T19:11:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7fbbc27fc640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbb0074500 0x7fbbb0076a10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbbc41581d0 tx=0x7fbbbc001360 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f41b3fff640 1 -- 172.21.15.28:0/1556001583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ac0de9b0 con 0x7f41ac0e2650 2024-04-25T19:11:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d80c8f30 con 0x7f59d80060e0 2024-04-25T19:11:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.167+0000 7f419b7fe640 1 -- 172.21.15.28:0/1556001583 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a8043070 con 0x7f41ac0e2650 2024-04-25T19:11:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.167+0000 7f419b7fe640 1 -- 172.21.15.28:0/1556001583 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a8030d00 con 0x7f41ac0e2650 2024-04-25T19:11:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.165+0000 7f90b67fc640 1 -- 172.21.15.28:0/3824554104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f90b000fcc0 con 0x7f90b81809c0 2024-04-25T19:11:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f90b67fc640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f90a40744b0 0x7f90a40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7f90b67fc640 1 -- 172.21.15.28:0/3824554104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f90b0095420 con 0x7f90b81809c0 2024-04-25T19:11:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.166+0000 7fc217fff640 1 -- 172.21.15.28:0/122950807 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc224018020 con 0x7fc2300cd880 2024-04-25T19:11:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.167+0000 7fc217fff640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc21c074580 0x7fc21c076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.167+0000 7fc217fff640 1 -- 172.21.15.28:0/122950807 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc2240957c0 con 0x7fc2300cd880 2024-04-25T19:11:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.167+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d80054a0 con 0x7f59d80060e0 2024-04-25T19:11:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.167+0000 7f90c5adb640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f90a40744b0 0x7f90a40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.167+0000 7fc22f7fe640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc21c074580 0x7fc21c076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.167+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90a8005290 con 0x7f90b81809c0 2024-04-25T19:11:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.168+0000 7f41b3fff640 1 -- 172.21.15.28:0/1556001583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ac0054a0 con 0x7f41ac0e2650 2024-04-25T19:11:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.168+0000 7f90c5adb640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f90a40744b0 0x7f90a40769c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f90bc007f80 tx=0x7f90bc009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.168+0000 7fc22f7fe640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc21c074580 0x7fc21c076a90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc2300afd40 tx=0x7fc228015040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.168+0000 7f419b7fe640 1 -- 172.21.15.28:0/1556001583 <== mon.1 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f41a8048460 con 0x7f41ac0e2650 2024-04-25T19:11:55.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.169+0000 7f419b7fe640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f41a00744e0 0x7f41a00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.169+0000 7f41b2ffd640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f41a00744e0 0x7f41a00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.169+0000 7f419b7fe640 1 -- 172.21.15.28:0/1556001583 <== mon.1 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f41a80b6370 con 0x7f41ac0e2650 2024-04-25T19:11:55.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.169+0000 7fa12effd640 1 -- 172.21.15.28:0/2633943184 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa134017070 con 0x7fa1400b2460 2024-04-25T19:11:55.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.169+0000 7fa13f59e640 1 -- 172.21.15.28:0/2633943184 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1401eef10 con 0x7fa1400b2460 2024-04-25T19:11:55.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.169+0000 7f915effd640 1 -- 172.21.15.28:0/2426175603 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160003b90 con 0x7f916815e220 2024-04-25T19:11:55.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.170+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f916814d250 con 0x7f916815e220 2024-04-25T19:11:55.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.169+0000 7fa13f59e640 1 -- 172.21.15.28:0/2633943184 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1400ce720 con 0x7fa1400b2460 2024-04-25T19:11:55.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.170+0000 7f41b2ffd640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f41a00744e0 0x7f41a00769f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f41a4006d40 tx=0x7f41a4017040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.170+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f916814ff80 con 0x7f916815e220 2024-04-25T19:11:55.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.169+0000 7fc217fff640 1 -- 172.21.15.28:0/122950807 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc22401f020 con 0x7fc2300cd880 2024-04-25T19:11:55.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.171+0000 7fa12effd640 1 -- 172.21.15.28:0/2633943184 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa134005990 con 0x7fa1400b2460 2024-04-25T19:11:55.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.171+0000 7fa12effd640 1 -- 172.21.15.28:0/2633943184 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa134021060 con 0x7fa1400b2460 2024-04-25T19:11:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.170+0000 7f915effd640 1 -- 172.21.15.28:0/2426175603 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9160003d30 con 0x7f916815e220 2024-04-25T19:11:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.170+0000 7f915effd640 1 -- 172.21.15.28:0/2426175603 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9160039880 con 0x7f916815e220 2024-04-25T19:11:55.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.172+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9168050740 con 0x7f916815e220 2024-04-25T19:11:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.172+0000 7f59dcff9640 1 -- 172.21.15.28:0/1155374226 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f59d403c6e0 con 0x7f59d80060e0 2024-04-25T19:11:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.172+0000 7f59dcff9640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f59c4074580 0x7f59c4076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.172+0000 7fa12effd640 1 -- 172.21.15.28:0/2633943184 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa13401a070 con 0x7fa1400b2460 2024-04-25T19:11:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.173+0000 7f59dcff9640 1 -- 172.21.15.28:0/1155374226 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f59d40b61d0 con 0x7f59d80060e0 2024-04-25T19:11:55.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.172+0000 7fa12cff9640 1 -- 172.21.15.28:0/2633943184 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa118005290 con 0x7fa1400b2460 2024-04-25T19:11:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.173+0000 7f59df7fe640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f59c4074580 0x7f59c4076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.173+0000 7fa12effd640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.173+0000 7fa13e59c640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.173+0000 7fa13e59c640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa130009d50 tx=0x7fa1300090e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.173+0000 7f59df7fe640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f59c4074580 0x7f59c4076a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f59d80d87d0 tx=0x7f59cc000f10 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.174+0000 7fbbab7fe640 1 -- 172.21.15.28:0/3285888950 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbbb4007210 con 0x7fbbac003a00 2024-04-25T19:11:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.175+0000 7fa12effd640 1 -- 172.21.15.28:0/2633943184 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa134093490 con 0x7fa1400b2460 2024-04-25T19:11:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.175+0000 7f419b7fe640 1 -- 172.21.15.28:0/1556001583 <== mon.1 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41a80801e0 con 0x7f41ac0e2650 2024-04-25T19:11:55.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.175+0000 7f59dcff9640 1 -- 172.21.15.28:0/1155374226 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f59d80cb730 con 0x7f59d80060e0 2024-04-25T19:11:55.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.177+0000 7f915effd640 1 -- 172.21.15.28:0/2426175603 <== mon.2 v2:172.21.15.53:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f916004a060 con 0x7f916815e220 2024-04-25T19:11:55.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.178+0000 7f915effd640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f914c0744b0 0x7f914c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.178+0000 7f916e0fa640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f914c0744b0 0x7f914c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.179+0000 7f916e0fa640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f914c0744b0 0x7f914c0769c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9158007c10 tx=0x7f9158007b60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.180+0000 7f915effd640 1 -- 172.21.15.28:0/2426175603 <== mon.2 v2:172.21.15.53:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9160031700 con 0x7f916815e220 2024-04-25T19:11:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.180+0000 7fa12effd640 1 -- 172.21.15.28:0/2633943184 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa13405f0b0 con 0x7fa1400b2460 2024-04-25T19:11:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.185+0000 7f90b67fc640 1 -- 172.21.15.28:0/3824554104 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f90b005f210 con 0x7f90b81809c0 2024-04-25T19:11:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.186+0000 7f915effd640 1 -- 172.21.15.28:0/2426175603 <== mon.2 v2:172.21.15.53:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9160051080 con 0x7f916815e220 2024-04-25T19:11:55.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.598+0000 7fc2349ef640 1 -- 172.21.15.28:0/122950807 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc230070fc0 con 0x7fc2300cd880 2024-04-25T19:11:55.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.599+0000 7fc217fff640 1 -- 172.21.15.28:0/122950807 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc22405f630 con 0x7fc2300cd880 2024-04-25T19:11:55.600 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.601+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb21c050740 con 0x7fb21c0aeb20 2024-04-25T19:11:55.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fc215ffb640 1 -- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc21c074580 msgr2=0x7fc21c076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fc215ffb640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc21c074580 0x7fc21c076a90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc2300afd40 tx=0x7fc228015040 comp rx=0 tx=0).stop 2024-04-25T19:11:55.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fb203fff640 1 -- 172.21.15.28:0/1494504966 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb214010040 con 0x7fb21c0aeb20 2024-04-25T19:11:55.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fc215ffb640 1 -- 172.21.15.28:0/122950807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 msgr2=0x7fc23008b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.604 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fc215ffb640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc23008b4a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc22400b3e0 tx=0x7fc22400b8a0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fc215ffb640 1 -- 172.21.15.28:0/122950807 shutdown_connections 2024-04-25T19:11:55.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fc215ffb640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc21c074580 0x7fc21c076a90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fc215ffb640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc2300cd880 0x7fc23008b4a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.602+0000 7fc215ffb640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2300b0dc0 0x7fc230166420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.603+0000 7fc215ffb640 1 --2- 172.21.15.28:0/122950807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2300ae8f0 0x7fc230207440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.603+0000 7fc215ffb640 1 -- 172.21.15.28:0/122950807 >> 172.21.15.28:0/122950807 conn(0x7fc2300740f0 msgr2=0x7fc2300ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.603+0000 7fc215ffb640 1 -- 172.21.15.28:0/122950807 shutdown_connections 2024-04-25T19:11:55.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.603+0000 7fc215ffb640 1 -- 172.21.15.28:0/122950807 wait complete. 2024-04-25T19:11:55.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.603+0000 7f41b3fff640 1 -- 172.21.15.28:0/1556001583 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f41ac0ddea0 con 0x7f41ac0e2650 2024-04-25T19:11:55.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.604+0000 7f419b7fe640 1 -- 172.21.15.28:0/1556001583 <== mon.1 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f41a80844e0 con 0x7f41ac0e2650 2024-04-25T19:11:55.607 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-04-25T19:11:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.605+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2080745b0 msgr2=0x7fb208076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.605+0000 7fb221a15640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2080745b0 0x7fb208076ac0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb20c007a40 tx=0x7fb20c008040 comp rx=0 tx=0).stop 2024-04-25T19:11:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.606+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c0aeb20 msgr2=0x7fb21c14f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.606+0000 7fb221a15640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c0aeb20 0x7fb21c14f410 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb21400eba0 tx=0x7fb21400c490 comp rx=0 tx=0).stop 2024-04-25T19:11:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.606+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 shutdown_connections 2024-04-25T19:11:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.606+0000 7fb221a15640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb2080745b0 0x7fb208076ac0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.606+0000 7fb221a15640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb21c15d0d0 0x7fb21c09dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.606+0000 7fb221a15640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb21c0aeb20 0x7fb21c14f410 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.606+0000 7fb221a15640 1 --2- 172.21.15.28:0/1494504966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb21c06dc90 0x7fb21c153a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.606+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 >> 172.21.15.28:0/1494504966 conn(0x7fb21c074310 msgr2=0x7fb21c0b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.607+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 shutdown_connections 2024-04-25T19:11:55.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.607+0000 7fb221a15640 1 -- 172.21.15.28:0/1494504966 wait complete. 2024-04-25T19:11:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 -- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f41a00744e0 msgr2=0x7f41a00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f41a00744e0 0x7f41a00769f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f41a4006d40 tx=0x7f41a4017040 comp rx=0 tx=0).stop 2024-04-25T19:11:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 -- 172.21.15.28:0/1556001583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41ac0e2650 msgr2=0x7f41ac0dd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41ac0e2650 0x7f41ac0dd390 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f41ac006390 tx=0x7f41a80046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 -- 172.21.15.28:0/1556001583 shutdown_connections 2024-04-25T19:11:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f41a00744e0 0x7f41a00769f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41ac0e5180 0x7f41ac0dd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41ac0e2650 0x7f41ac0dd390 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 --2- 172.21.15.28:0/1556001583 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f41ac0522f0 0x7f41ac0d8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 -- 172.21.15.28:0/1556001583 >> 172.21.15.28:0/1556001583 conn(0x7f41ac00c3f0 msgr2=0x7f41ac0e1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 -- 172.21.15.28:0/1556001583 shutdown_connections 2024-04-25T19:11:55.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.608+0000 7f41997fa640 1 -- 172.21.15.28:0/1556001583 wait complete. 2024-04-25T19:11:55.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.614+0000 7fa12cff9640 1 -- 172.21.15.28:0/2633943184 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa118005580 con 0x7fa1400b2460 2024-04-25T19:11:55.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.615+0000 7fa12effd640 1 -- 172.21.15.28:0/2633943184 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa1340633b0 con 0x7fa1400b2460 2024-04-25T19:11:55.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.615+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbbc40710c0 con 0x7fbbac003a00 2024-04-25T19:11:55.617 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:55.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.616+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f59d8006750 con 0x7f59d80060e0 2024-04-25T19:11:55.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.616+0000 7fbbab7fe640 1 -- 172.21.15.28:0/3285888950 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbbb4007210 con 0x7fbbac003a00 2024-04-25T19:11:55.618 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:55.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.617+0000 7f59dcff9640 1 -- 172.21.15.28:0/1155374226 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f59d4042020 con 0x7f59d80060e0 2024-04-25T19:11:55.619 INFO:teuthology.orchestra.run.smithi028.stdout:55834574857 2024-04-25T19:11:55.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.618+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f90a8005580 con 0x7f90b81809c0 2024-04-25T19:11:55.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.619+0000 7f90b67fc640 1 -- 172.21.15.28:0/3824554104 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f90b0063510 con 0x7f90b81809c0 2024-04-25T19:11:55.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.619+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f91680cf520 con 0x7f916815e220 2024-04-25T19:11:55.621 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.619+0000 7fa12cff9640 1 -- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 msgr2=0x7fa1200769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.619+0000 7fa12cff9640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa130009d50 tx=0x7fa1300090e0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.619+0000 7fa12cff9640 1 -- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400b2460 msgr2=0x7fa1400d1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.619+0000 7fa12cff9640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400b2460 0x7fa1400d1300 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa13400e8a0 tx=0x7fa13400ed60 comp rx=0 tx=0).stop 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.619+0000 7fa12cff9640 1 -- 172.21.15.28:0/2633943184 shutdown_connections 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.619+0000 7fa12cff9640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7fa12cff9640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1400ce070 0x7fa140150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7fa12cff9640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa1400b2460 0x7fa1400d1300 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7fa12cff9640 1 --2- 172.21.15.28:0/2633943184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1400af570 0x7fa1401ee720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7fa12cff9640 1 -- 172.21.15.28:0/2633943184 >> 172.21.15.28:0/2633943184 conn(0x7fa140073b80 msgr2=0x7fa14006d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7fa12cff9640 1 -- 172.21.15.28:0/2633943184 shutdown_connections 2024-04-25T19:11:55.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7fa12cff9640 1 -- 172.21.15.28:0/2633943184 wait complete. 2024-04-25T19:11:55.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7f915effd640 1 -- 172.21.15.28:0/2426175603 <== mon.2 v2:172.21.15.53:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f916003e070 con 0x7f916815e220 2024-04-25T19:11:55.623 INFO:teuthology.orchestra.run.smithi028.stdout:55834574858 2024-04-25T19:11:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f59c4074580 msgr2=0x7f59c4076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7f59e54fc640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f59c4074580 0x7f59c4076a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f59d80d87d0 tx=0x7f59cc000f10 comp rx=0 tx=0).stop 2024-04-25T19:11:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59d80060e0 msgr2=0x7f59d80d9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59d80060e0 0x7f59d80d9180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f59d8007cb0 tx=0x7f59d40046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbb0074500 msgr2=0x7fbbb0076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.620+0000 7fbbc9696640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbb0074500 0x7fbbb0076a10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbbc41581d0 tx=0x7fbbbc001360 comp rx=0 tx=0).stop 2024-04-25T19:11:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 msgr2=0x7fbbc4144210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbc4144210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40714e0 tx=0x7fbbb4001120 comp rx=0 tx=0).stop 2024-04-25T19:11:55.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 shutdown_connections 2024-04-25T19:11:55.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f59c4074580 0x7f59c4076a90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d80c81c0 0x7f59d80c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59d8037bb0 0x7f59d80cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 --2- 172.21.15.28:0/1155374226 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f59d80060e0 0x7f59d80d9180 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 >> 172.21.15.28:0/1155374226 conn(0x7f59d8054cf0 msgr2=0x7f59d80349a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 shutdown_connections 2024-04-25T19:11:55.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbb0074500 0x7fbbb0076a10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc415bc50 0x7fbbc4143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc415b840 0x7fbbc41569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 --2- 172.21.15.28:0/3285888950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbac003a00 0x7fbbc4144210 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 >> 172.21.15.28:0/3285888950 conn(0x7fbbc40742c0 msgr2=0x7fbbc40b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 shutdown_connections 2024-04-25T19:11:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7fbbc9696640 1 -- 172.21.15.28:0/3285888950 wait complete. 2024-04-25T19:11:55.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.621+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 shutdown_connections 2024-04-25T19:11:55.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.622+0000 7f59e54fc640 1 -- 172.21.15.28:0/1155374226 wait complete. 2024-04-25T19:11:55.628 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-04-25T19:11:55.629 DEBUG:teuthology.parallel:result is None 2024-04-25T19:11:55.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f90a40744b0 msgr2=0x7f90a40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f90a40744b0 0x7f90a40769c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f90bc007f80 tx=0x7f90bc009030 comp rx=0 tx=0).stop 2024-04-25T19:11:55.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90b81809c0 msgr2=0x7f90b8180dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90b81809c0 0x7f90b8180dd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f90b0001270 tx=0x7f90b0007430 comp rx=0 tx=0).stop 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 shutdown_connections 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f90a40744b0 0x7f90a40769c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90b80e5240 0x7f90b80e5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90b81809c0 0x7f90b8180dd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 --2- 172.21.15.28:0/3824554104 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f90ac004c00 0x7f90b8171b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.623+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 >> 172.21.15.28:0/3824554104 conn(0x7f90b8054cf0 msgr2=0x7f90b8043f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 shutdown_connections 2024-04-25T19:11:55.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f90c7d3f640 1 -- 172.21.15.28:0/3824554104 wait complete. 2024-04-25T19:11:55.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f914c0744b0 msgr2=0x7f914c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f914c0744b0 0x7f914c0769c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9158007c10 tx=0x7f9158007b60 comp rx=0 tx=0).stop 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916815e220 msgr2=0x7f9168154560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916815e220 0x7f9168154560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9160030000 tx=0x7f9160004a40 comp rx=0 tx=0).stop 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 shutdown_connections 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f914c0744b0 0x7f914c0769c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f916815e220 0x7f9168154560 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91680cfa00 0x7f91681525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 --2- 172.21.15.28:0/2426175603 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f916809b540 0x7f916814bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 >> 172.21.15.28:0/2426175603 conn(0x7f91680b7620 msgr2=0x7f9168071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 shutdown_connections 2024-04-25T19:11:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.624+0000 7f917035e640 1 -- 172.21.15.28:0/2426175603 wait complete. 2024-04-25T19:11:55.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.2 2024-04-25T19:11:55.633 DEBUG:teuthology.parallel:result is None 2024-04-25T19:11:55.634 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-04-25T19:11:55.634 DEBUG:teuthology.parallel:result is None 2024-04-25T19:11:55.643 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-04-25T19:11:55.643 DEBUG:teuthology.parallel:result is None 2024-04-25T19:11:55.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2024-04-25T19:11:55.644 DEBUG:teuthology.parallel:result is None 2024-04-25T19:11:55.645 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.0 2024-04-25T19:11:55.645 DEBUG:teuthology.parallel:result is None 2024-04-25T19:11:55.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.4 2024-04-25T19:11:55.646 DEBUG:teuthology.parallel:result is None 2024-04-25T19:11:55.647 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2024-04-25T19:11:55.647 DEBUG:teuthology.parallel:result is None 2024-04-25T19:11:55.647 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T19:11:55.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:11:55.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.796+0000 7f60b5bb2640 1 Processor -- start 2024-04-25T19:11:55.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.796+0000 7f60b5bb2640 1 -- start start 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60b5bb2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b00b4e30 0x7f60b00bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60b5bb2640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f60b00b9470 0x7f60b00b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60b5bb2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b00bbb90 con 0x7f60b00b7650 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60b5bb2640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f60b00d0b50 con 0x7f60b00b9470 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60b5bb2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f60b008e3a0 con 0x7f60b00b4e30 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60affff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f60b00b7650 0x7f60b00b7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59014/0 (socket says 172.21.15.28:59014) 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60affff640 1 -- 172.21.15.28:0/52314081 learned_addr learned my addr 172.21.15.28:0/52314081 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60aeffd640 1 --2- 172.21.15.28:0/52314081 >> v2:172.21.15.53:3300/0 conn(0x7f60b00b9470 0x7f60b00b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60af7fe640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b00b4e30 0x7f60b00bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3442150252 0 0) 0x7f60a8001490 con 0x7f60b00b7650 2024-04-25T19:11:55.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6098003520 con 0x7f60b00b7650 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4041933778 0 0) 0x7f6098003520 con 0x7f60b00b7650 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b00b4e30 msgr2=0x7f60b00bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60ae7fc640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b00b4e30 0x7f60b00bb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 >> v2:172.21.15.53:3300/0 conn(0x7f60b00b9470 msgr2=0x7f60b00b9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.798+0000 7f60ae7fc640 1 --2- 172.21.15.28:0/52314081 >> v2:172.21.15.53:3300/0 conn(0x7f60b00b9470 0x7f60b00b9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b0164d60 con 0x7f60b00b7650 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f60a8001bc0 con 0x7f60b00b7650 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 >> v1:172.21.15.28:6789/0 conn(0x7f60b00b7650 legacy=0x7f60b00b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f6098003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6098004800 0x7f6098004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b00b4e30 0x7f6098005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60b00d0b50 con 0x7f60980039b0 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f60b008e3a0 con 0x7f6098004800 2024-04-25T19:11:55.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f609800ba70 con 0x7f60b00b4e30 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60af7fe640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f6098003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60aeffd640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6098004800 0x7f6098004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60affff640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b00b4e30 0x7f6098005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60af7fe640 1 -- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b00b4e30 msgr2=0x7f6098005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60af7fe640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b00b4e30 0x7f6098005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60af7fe640 1 -- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6098004800 msgr2=0x7f6098004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60af7fe640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6098004800 0x7f6098004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.799+0000 7f60af7fe640 1 -- 172.21.15.28:0/52314081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6098003520 con 0x7f60980039b0 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.800+0000 7f60af7fe640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f6098003e20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009a90 tx=0x7f6098005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac88e6c1edaab83f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.800+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0036070 con 0x7f60980039b0 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.800+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a0031400 con 0x7f60980039b0 2024-04-25T19:11:55.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.800+0000 7f60ae7fc640 1 -- 172.21.15.28:0/52314081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0009690 con 0x7f60980039b0 2024-04-25T19:11:55.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.801+0000 7f60b5bb2640 1 -- 172.21.15.28:0/52314081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 msgr2=0x7f6098003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.801+0000 7f60b5bb2640 1 --2- 172.21.15.28:0/52314081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f6098003e20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009a90 tx=0x7f6098005830 comp rx=0 tx=0).stop 2024-04-25T19:11:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.801+0000 7f60affff640 1 -- 172.21.15.28:0/52314081 reap_dead start 2024-04-25T19:11:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.801+0000 7f60b5bb2640 1 -- 172.21.15.28:0/52314081 shutdown_connections 2024-04-25T19:11:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.801+0000 7f60b5bb2640 1 -- 172.21.15.28:0/52314081 >> 172.21.15.28:0/52314081 conn(0x7f60b00d2650 msgr2=0x7f60b00d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.801+0000 7f60b5bb2640 1 -- 172.21.15.28:0/52314081 shutdown_connections 2024-04-25T19:11:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.801+0000 7f60b5bb2640 1 -- 172.21.15.28:0/52314081 wait complete. 2024-04-25T19:11:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.802+0000 7f60b5bb2640 1 Processor -- start 2024-04-25T19:11:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.803+0000 7f60b5bb2640 1 -- start start 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60b5bb2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f60b0144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60b5bb2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b015c4a0 0x7f60b0162c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60b5bb2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b015c8b0 0x7f60b01631a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60b5bb2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60b00d07e0 con 0x7f60980039b0 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60b5bb2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f60b00d0500 con 0x7f60b015c4a0 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60b5bb2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f60b01649c0 con 0x7f60b015c8b0 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f60b0144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f60b0144e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37360/0 (socket says 172.21.15.28:37360) 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60af7fe640 1 -- 172.21.15.28:0/491546489 learned_addr learned my addr 172.21.15.28:0/491546489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60aeffd640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b015c4a0 0x7f60b0162c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60af7fe640 1 -- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b015c8b0 msgr2=0x7f60b01631a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60affff640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b015c8b0 0x7f60b01631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60af7fe640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b015c8b0 0x7f60b01631a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60af7fe640 1 -- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b015c4a0 msgr2=0x7f60b0162c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60af7fe640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b015c4a0 0x7f60b0162c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60af7fe640 1 -- 172.21.15.28:0/491546489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b015b400 con 0x7f60980039b0 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.804+0000 7f60aeffd640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b015c4a0 0x7f60b0162c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.805+0000 7f60af7fe640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f60b0144e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009bb0 tx=0x7f60a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.805+0000 7f6097fff640 1 -- 172.21.15.28:0/491546489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0040070 con 0x7f60980039b0 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.805+0000 7f6097fff640 1 -- 172.21.15.28:0/491546489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a003b460 con 0x7f60980039b0 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.805+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b015b6e0 con 0x7f60980039b0 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.805+0000 7f6097fff640 1 -- 172.21.15.28:0/491546489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a0036040 con 0x7f60980039b0 2024-04-25T19:11:55.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.805+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b015bca0 con 0x7f60980039b0 2024-04-25T19:11:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.806+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b0050740 con 0x7f60980039b0 2024-04-25T19:11:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.807+0000 7f6097fff640 1 -- 172.21.15.28:0/491546489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f60a0043070 con 0x7f60980039b0 2024-04-25T19:11:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.807+0000 7f6097fff640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f609c0744b0 0x7f609c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.807+0000 7f6097fff640 1 -- 172.21.15.28:0/491546489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f60a00b5ed0 con 0x7f60980039b0 2024-04-25T19:11:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.807+0000 7f60aeffd640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f609c0744b0 0x7f609c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.808+0000 7f60aeffd640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f609c0744b0 0x7f609c0769c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f60a40090e0 tx=0x7f60a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:55.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:55.812+0000 7f6097fff640 1 -- 172.21.15.28:0/491546489 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60a007fc50 con 0x7f60980039b0 2024-04-25T19:11:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.002+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f60b015af40 con 0x7f609c0744b0 2024-04-25T19:11:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.010+0000 7f6097fff640 1 -- 172.21.15.28:0/491546489 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332522 (secure 0 0 0) 0x7f60a40129e0 con 0x7f609c0744b0 2024-04-25T19:11:56.016 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:56.018 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:11:56.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f609c0744b0 msgr2=0x7f609c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f609c0744b0 0x7f609c0769c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f60a40090e0 tx=0x7f60a4008040 comp rx=0 tx=0).stop 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 msgr2=0x7f60b0144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f60b0144e10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f60a0009bb0 tx=0x7f60a0004730 comp rx=0 tx=0).stop 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 shutdown_connections 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f609c0744b0 0x7f609c0769c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f60b015c8b0 0x7f60b01631a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b015c4a0 0x7f60b0162c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 --2- 172.21.15.28:0/491546489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60980039b0 0x7f60b0144e10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.014+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 >> 172.21.15.28:0/491546489 conn(0x7f60b00d2650 msgr2=0x7f60b00b6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.015+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 shutdown_connections 2024-04-25T19:11:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.015+0000 7f60b5bb2640 1 -- 172.21.15.28:0/491546489 wait complete. 2024-04-25T19:11:56.028 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-25T19:11:54.842514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252656,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722064,"statfs":{"total":773094113280,"available":772835393536,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003591"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878592+0000","last_change":"2024-04-25T19:11:28.938792+0000","last_active":"2024-04-25T19:11:35.878592+0000","last_peered":"2024-04-25T19:11:35.878592+0000","last_clean":"2024-04-25T19:11:35.878592+0000","last_became_active":"2024-04-25T19:11:28.938417+0000","last_became_peered":"2024-04-25T19:11:28.938417+0000","last_unstale":"2024-04-25T19:11:35.878592+0000","last_undegraded":"2024-04-25T19:11:35.878592+0000","last_fullsized":"2024-04-25T19:11:35.878592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:44.780178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171775+0000","last_change":"2024-04-25T19:11:29.937823+0000","last_active":"2024-04-25T19:11:37.171775+0000","last_peered":"2024-04-25T19:11:37.171775+0000","last_clean":"2024-04-25T19:11:37.171775+0000","last_became_active":"2024-04-25T19:11:29.937501+0000","last_became_peered":"2024-04-25T19:11:29.937501+0000","last_unstale":"2024-04-25T19:11:37.171775+0000","last_undegraded":"2024-04-25T19:11:37.171775+0000","last_fullsized":"2024-04-25T19:11:37.171775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170976+0000","last_change":"2024-04-25T19:11:34.113864+0000","last_active":"2024-04-25T19:11:37.170976+0000","last_peered":"2024-04-25T19:11:37.170976+0000","last_clean":"2024-04-25T19:11:37.170976+0000","last_became_active":"2024-04-25T19:11:34.113449+0000","last_became_peered":"2024-04-25T19:11:34.113449+0000","last_unstale":"2024-04-25T19:11:37.170976+0000","last_undegraded":"2024-04-25T19:11:37.170976+0000","last_fullsized":"2024-04-25T19:11:37.170976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.247905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170712+0000","last_change":"2024-04-25T19:11:28.934472+0000","last_active":"2024-04-25T19:11:37.170712+0000","last_peered":"2024-04-25T19:11:37.170712+0000","last_clean":"2024-04-25T19:11:37.170712+0000","last_became_active":"2024-04-25T19:11:28.934016+0000","last_became_peered":"2024-04-25T19:11:28.934016+0000","last_unstale":"2024-04-25T19:11:37.170712+0000","last_undegraded":"2024-04-25T19:11:37.170712+0000","last_fullsized":"2024-04-25T19:11:37.170712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:27:30.904833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878257+0000","last_change":"2024-04-25T19:11:29.944065+0000","last_active":"2024-04-25T19:11:35.878257+0000","last_peered":"2024-04-25T19:11:35.878257+0000","last_clean":"2024-04-25T19:11:35.878257+0000","last_became_active":"2024-04-25T19:11:29.943773+0000","last_became_peered":"2024-04-25T19:11:29.943773+0000","last_unstale":"2024-04-25T19:11:35.878257+0000","last_undegraded":"2024-04-25T19:11:35.878257+0000","last_fullsized":"2024-04-25T19:11:35.878257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:15.552217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989732+0000","last_change":"2024-04-25T19:11:33.975423+0000","last_active":"2024-04-25T19:11:36.989732+0000","last_peered":"2024-04-25T19:11:36.989732+0000","last_clean":"2024-04-25T19:11:36.989732+0000","last_became_active":"2024-04-25T19:11:33.974878+0000","last_became_peered":"2024-04-25T19:11:33.974878+0000","last_unstale":"2024-04-25T19:11:36.989732+0000","last_undegraded":"2024-04-25T19:11:36.989732+0000","last_fullsized":"2024-04-25T19:11:36.989732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:23.927981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170963+0000","last_change":"2024-04-25T19:11:28.937102+0000","last_active":"2024-04-25T19:11:37.170963+0000","last_peered":"2024-04-25T19:11:37.170963+0000","last_clean":"2024-04-25T19:11:37.170963+0000","last_became_active":"2024-04-25T19:11:28.936699+0000","last_became_peered":"2024-04-25T19:11:28.936699+0000","last_unstale":"2024-04-25T19:11:37.170963+0000","last_undegraded":"2024-04-25T19:11:37.170963+0000","last_fullsized":"2024-04-25T19:11:37.170963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:38.236609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962397+0000","last_change":"2024-04-25T19:11:29.947706+0000","last_active":"2024-04-25T19:11:34.962397+0000","last_peered":"2024-04-25T19:11:34.962397+0000","last_clean":"2024-04-25T19:11:34.962397+0000","last_became_active":"2024-04-25T19:11:29.945963+0000","last_became_peered":"2024-04-25T19:11:29.945963+0000","last_unstale":"2024-04-25T19:11:34.962397+0000","last_undegraded":"2024-04-25T19:11:34.962397+0000","last_fullsized":"2024-04-25T19:11:34.962397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:51.175553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990390+0000","last_change":"2024-04-25T19:11:33.975724+0000","last_active":"2024-04-25T19:11:36.990390+0000","last_peered":"2024-04-25T19:11:36.990390+0000","last_clean":"2024-04-25T19:11:36.990390+0000","last_became_active":"2024-04-25T19:11:33.974921+0000","last_became_peered":"2024-04-25T19:11:33.974921+0000","last_unstale":"2024-04-25T19:11:36.990390+0000","last_undegraded":"2024-04-25T19:11:36.990390+0000","last_fullsized":"2024-04-25T19:11:36.990390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:55.208890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170918+0000","last_change":"2024-04-25T19:11:28.940100+0000","last_active":"2024-04-25T19:11:37.170918+0000","last_peered":"2024-04-25T19:11:37.170918+0000","last_clean":"2024-04-25T19:11:37.170918+0000","last_became_active":"2024-04-25T19:11:28.939482+0000","last_became_peered":"2024-04-25T19:11:28.939482+0000","last_unstale":"2024-04-25T19:11:37.170918+0000","last_undegraded":"2024-04-25T19:11:37.170918+0000","last_fullsized":"2024-04-25T19:11:37.170918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962428+0000","last_change":"2024-04-25T19:11:29.945900+0000","last_active":"2024-04-25T19:11:34.962428+0000","last_peered":"2024-04-25T19:11:34.962428+0000","last_clean":"2024-04-25T19:11:34.962428+0000","last_became_active":"2024-04-25T19:11:29.945554+0000","last_became_peered":"2024-04-25T19:11:29.945554+0000","last_unstale":"2024-04-25T19:11:34.962428+0000","last_undegraded":"2024-04-25T19:11:34.962428+0000","last_fullsized":"2024-04-25T19:11:34.962428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990011+0000","last_change":"2024-04-25T19:11:33.975263+0000","last_active":"2024-04-25T19:11:36.990011+0000","last_peered":"2024-04-25T19:11:36.990011+0000","last_clean":"2024-04-25T19:11:36.990011+0000","last_became_active":"2024-04-25T19:11:33.974661+0000","last_became_peered":"2024-04-25T19:11:33.974661+0000","last_unstale":"2024-04-25T19:11:36.990011+0000","last_undegraded":"2024-04-25T19:11:36.990011+0000","last_fullsized":"2024-04-25T19:11:36.990011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:29.019166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961851+0000","last_change":"2024-04-25T19:11:29.939828+0000","last_active":"2024-04-25T19:11:34.961851+0000","last_peered":"2024-04-25T19:11:34.961851+0000","last_clean":"2024-04-25T19:11:34.961851+0000","last_became_active":"2024-04-25T19:11:29.939513+0000","last_became_peered":"2024-04-25T19:11:29.939513+0000","last_unstale":"2024-04-25T19:11:34.961851+0000","last_undegraded":"2024-04-25T19:11:34.961851+0000","last_fullsized":"2024-04-25T19:11:34.961851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171950+0000","last_change":"2024-04-25T19:11:28.937339+0000","last_active":"2024-04-25T19:11:37.171950+0000","last_peered":"2024-04-25T19:11:37.171950+0000","last_clean":"2024-04-25T19:11:37.171950+0000","last_became_active":"2024-04-25T19:11:28.936962+0000","last_became_peered":"2024-04-25T19:11:28.936962+0000","last_unstale":"2024-04-25T19:11:37.171950+0000","last_undegraded":"2024-04-25T19:11:37.171950+0000","last_fullsized":"2024-04-25T19:11:37.171950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:30.900062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990657+0000","last_change":"2024-04-25T19:11:33.975365+0000","last_active":"2024-04-25T19:11:36.990657+0000","last_peered":"2024-04-25T19:11:36.990657+0000","last_clean":"2024-04-25T19:11:36.990657+0000","last_became_active":"2024-04-25T19:11:33.974761+0000","last_became_peered":"2024-04-25T19:11:33.974761+0000","last_unstale":"2024-04-25T19:11:36.990657+0000","last_undegraded":"2024-04-25T19:11:36.990657+0000","last_fullsized":"2024-04-25T19:11:36.990657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:46.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171339+0000","last_change":"2024-04-25T19:11:29.935255+0000","last_active":"2024-04-25T19:11:37.171339+0000","last_peered":"2024-04-25T19:11:37.171339+0000","last_clean":"2024-04-25T19:11:37.171339+0000","last_became_active":"2024-04-25T19:11:29.934911+0000","last_became_peered":"2024-04-25T19:11:29.934911+0000","last_unstale":"2024-04-25T19:11:37.171339+0000","last_undegraded":"2024-04-25T19:11:37.171339+0000","last_fullsized":"2024-04-25T19:11:37.171339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470651+0000","last_change":"2024-04-25T19:11:28.937507+0000","last_active":"2024-04-25T19:11:35.470651+0000","last_peered":"2024-04-25T19:11:35.470651+0000","last_clean":"2024-04-25T19:11:35.470651+0000","last_became_active":"2024-04-25T19:11:28.936655+0000","last_became_peered":"2024-04-25T19:11:28.936655+0000","last_unstale":"2024-04-25T19:11:35.470651+0000","last_undegraded":"2024-04-25T19:11:35.470651+0000","last_fullsized":"2024-04-25T19:11:35.470651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990392+0000","last_change":"2024-04-25T19:11:33.974529+0000","last_active":"2024-04-25T19:11:36.990392+0000","last_peered":"2024-04-25T19:11:36.990392+0000","last_clean":"2024-04-25T19:11:36.990392+0000","last_became_active":"2024-04-25T19:11:33.973907+0000","last_became_peered":"2024-04-25T19:11:33.973907+0000","last_unstale":"2024-04-25T19:11:36.990392+0000","last_undegraded":"2024-04-25T19:11:36.990392+0000","last_fullsized":"2024-04-25T19:11:36.990392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:52.041722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171394+0000","last_change":"2024-04-25T19:11:29.930361+0000","last_active":"2024-04-25T19:11:37.171394+0000","last_peered":"2024-04-25T19:11:37.171394+0000","last_clean":"2024-04-25T19:11:37.171394+0000","last_became_active":"2024-04-25T19:11:29.930060+0000","last_became_peered":"2024-04-25T19:11:29.930060+0000","last_unstale":"2024-04-25T19:11:37.171394+0000","last_undegraded":"2024-04-25T19:11:37.171394+0000","last_fullsized":"2024-04-25T19:11:37.171394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470271+0000","last_change":"2024-04-25T19:11:28.930410+0000","last_active":"2024-04-25T19:11:35.470271+0000","last_peered":"2024-04-25T19:11:35.470271+0000","last_clean":"2024-04-25T19:11:35.470271+0000","last_became_active":"2024-04-25T19:11:28.930012+0000","last_became_peered":"2024-04-25T19:11:28.930012+0000","last_unstale":"2024-04-25T19:11:35.470271+0000","last_undegraded":"2024-04-25T19:11:35.470271+0000","last_fullsized":"2024-04-25T19:11:35.470271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.471102+0000","last_change":"2024-04-25T19:11:34.115931+0000","last_active":"2024-04-25T19:11:35.471102+0000","last_peered":"2024-04-25T19:11:35.471102+0000","last_clean":"2024-04-25T19:11:35.471102+0000","last_became_active":"2024-04-25T19:11:34.115519+0000","last_became_peered":"2024-04-25T19:11:34.115519+0000","last_unstale":"2024-04-25T19:11:35.471102+0000","last_undegraded":"2024-04-25T19:11:35.471102+0000","last_fullsized":"2024-04-25T19:11:35.471102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.795134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990163+0000","last_change":"2024-04-25T19:11:29.930630+0000","last_active":"2024-04-25T19:11:36.990163+0000","last_peered":"2024-04-25T19:11:36.990163+0000","last_clean":"2024-04-25T19:11:36.990163+0000","last_became_active":"2024-04-25T19:11:29.930314+0000","last_became_peered":"2024-04-25T19:11:29.930314+0000","last_unstale":"2024-04-25T19:11:36.990163+0000","last_undegraded":"2024-04-25T19:11:36.990163+0000","last_fullsized":"2024-04-25T19:11:36.990163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:44.955421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170921+0000","last_change":"2024-04-25T19:11:28.938687+0000","last_active":"2024-04-25T19:11:37.170921+0000","last_peered":"2024-04-25T19:11:37.170921+0000","last_clean":"2024-04-25T19:11:37.170921+0000","last_became_active":"2024-04-25T19:11:28.938288+0000","last_became_peered":"2024-04-25T19:11:28.938288+0000","last_unstale":"2024-04-25T19:11:37.170921+0000","last_undegraded":"2024-04-25T19:11:37.170921+0000","last_fullsized":"2024-04-25T19:11:37.170921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990666+0000","last_change":"2024-04-25T19:11:33.972056+0000","last_active":"2024-04-25T19:11:36.990666+0000","last_peered":"2024-04-25T19:11:36.990666+0000","last_clean":"2024-04-25T19:11:36.990666+0000","last_became_active":"2024-04-25T19:11:33.971484+0000","last_became_peered":"2024-04-25T19:11:33.971484+0000","last_unstale":"2024-04-25T19:11:36.990666+0000","last_undegraded":"2024-04-25T19:11:36.990666+0000","last_fullsized":"2024-04-25T19:11:36.990666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:52.951275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878644+0000","last_change":"2024-04-25T19:11:29.935281+0000","last_active":"2024-04-25T19:11:35.878644+0000","last_peered":"2024-04-25T19:11:35.878644+0000","last_clean":"2024-04-25T19:11:35.878644+0000","last_became_active":"2024-04-25T19:11:29.934928+0000","last_became_peered":"2024-04-25T19:11:29.934928+0000","last_unstale":"2024-04-25T19:11:35.878644+0000","last_undegraded":"2024-04-25T19:11:35.878644+0000","last_fullsized":"2024-04-25T19:11:35.878644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963243+0000","last_change":"2024-04-25T19:11:28.940436+0000","last_active":"2024-04-25T19:11:34.963243+0000","last_peered":"2024-04-25T19:11:34.963243+0000","last_clean":"2024-04-25T19:11:34.963243+0000","last_became_active":"2024-04-25T19:11:28.940050+0000","last_became_peered":"2024-04-25T19:11:28.940050+0000","last_unstale":"2024-04-25T19:11:34.963243+0000","last_undegraded":"2024-04-25T19:11:34.963243+0000","last_fullsized":"2024-04-25T19:11:34.963243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989678+0000","last_change":"2024-04-25T19:11:34.115813+0000","last_active":"2024-04-25T19:11:36.989678+0000","last_peered":"2024-04-25T19:11:36.989678+0000","last_clean":"2024-04-25T19:11:36.989678+0000","last_became_active":"2024-04-25T19:11:34.115268+0000","last_became_peered":"2024-04-25T19:11:34.115268+0000","last_unstale":"2024-04-25T19:11:36.989678+0000","last_undegraded":"2024-04-25T19:11:36.989678+0000","last_fullsized":"2024-04-25T19:11:36.989678+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:54.194127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171971+0000","last_change":"2024-04-25T19:11:29.945249+0000","last_active":"2024-04-25T19:11:37.171971+0000","last_peered":"2024-04-25T19:11:37.171971+0000","last_clean":"2024-04-25T19:11:37.171971+0000","last_became_active":"2024-04-25T19:11:29.944930+0000","last_became_peered":"2024-04-25T19:11:29.944930+0000","last_unstale":"2024-04-25T19:11:37.171971+0000","last_undegraded":"2024-04-25T19:11:37.171971+0000","last_fullsized":"2024-04-25T19:11:37.171971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470980+0000","last_change":"2024-04-25T19:11:28.937963+0000","last_active":"2024-04-25T19:11:35.470980+0000","last_peered":"2024-04-25T19:11:35.470980+0000","last_clean":"2024-04-25T19:11:35.470980+0000","last_became_active":"2024-04-25T19:11:28.936818+0000","last_became_peered":"2024-04-25T19:11:28.936818+0000","last_unstale":"2024-04-25T19:11:35.470980+0000","last_undegraded":"2024-04-25T19:11:35.470980+0000","last_fullsized":"2024-04-25T19:11:35.470980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170969+0000","last_change":"2024-04-25T19:11:34.113848+0000","last_active":"2024-04-25T19:11:37.170969+0000","last_peered":"2024-04-25T19:11:37.170969+0000","last_clean":"2024-04-25T19:11:37.170969+0000","last_became_active":"2024-04-25T19:11:34.113403+0000","last_became_peered":"2024-04-25T19:11:34.113403+0000","last_unstale":"2024-04-25T19:11:37.170969+0000","last_undegraded":"2024-04-25T19:11:37.170969+0000","last_fullsized":"2024-04-25T19:11:37.170969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:43.702771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470714+0000","last_change":"2024-04-25T19:11:29.930348+0000","last_active":"2024-04-25T19:11:35.470714+0000","last_peered":"2024-04-25T19:11:35.470714+0000","last_clean":"2024-04-25T19:11:35.470714+0000","last_became_active":"2024-04-25T19:11:29.930028+0000","last_became_peered":"2024-04-25T19:11:29.930028+0000","last_unstale":"2024-04-25T19:11:35.470714+0000","last_undegraded":"2024-04-25T19:11:35.470714+0000","last_fullsized":"2024-04-25T19:11:35.470714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170925+0000","last_change":"2024-04-25T19:11:28.939508+0000","last_active":"2024-04-25T19:11:37.170925+0000","last_peered":"2024-04-25T19:11:37.170925+0000","last_clean":"2024-04-25T19:11:37.170925+0000","last_became_active":"2024-04-25T19:11:28.939050+0000","last_became_peered":"2024-04-25T19:11:28.939050+0000","last_unstale":"2024-04-25T19:11:37.170925+0000","last_undegraded":"2024-04-25T19:11:37.170925+0000","last_fullsized":"2024-04-25T19:11:37.170925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:00.743906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878165+0000","last_change":"2024-04-25T19:11:34.110317+0000","last_active":"2024-04-25T19:11:35.878165+0000","last_peered":"2024-04-25T19:11:35.878165+0000","last_clean":"2024-04-25T19:11:35.878165+0000","last_became_active":"2024-04-25T19:11:34.109906+0000","last_became_peered":"2024-04-25T19:11:34.109906+0000","last_unstale":"2024-04-25T19:11:35.878165+0000","last_undegraded":"2024-04-25T19:11:35.878165+0000","last_fullsized":"2024-04-25T19:11:35.878165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:35.687485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962541+0000","last_change":"2024-04-25T19:11:29.952079+0000","last_active":"2024-04-25T19:11:34.962541+0000","last_peered":"2024-04-25T19:11:34.962541+0000","last_clean":"2024-04-25T19:11:34.962541+0000","last_became_active":"2024-04-25T19:11:29.951437+0000","last_became_peered":"2024-04-25T19:11:29.951437+0000","last_unstale":"2024-04-25T19:11:34.962541+0000","last_undegraded":"2024-04-25T19:11:34.962541+0000","last_fullsized":"2024-04-25T19:11:34.962541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470828+0000","last_change":"2024-04-25T19:11:28.933142+0000","last_active":"2024-04-25T19:11:35.470828+0000","last_peered":"2024-04-25T19:11:35.470828+0000","last_clean":"2024-04-25T19:11:35.470828+0000","last_became_active":"2024-04-25T19:11:28.932682+0000","last_became_peered":"2024-04-25T19:11:28.932682+0000","last_unstale":"2024-04-25T19:11:35.470828+0000","last_undegraded":"2024-04-25T19:11:35.470828+0000","last_fullsized":"2024-04-25T19:11:35.470828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:38.236609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470450+0000","last_change":"2024-04-25T19:11:33.971859+0000","last_active":"2024-04-25T19:11:35.470450+0000","last_peered":"2024-04-25T19:11:35.470450+0000","last_clean":"2024-04-25T19:11:35.470450+0000","last_became_active":"2024-04-25T19:11:33.971349+0000","last_became_peered":"2024-04-25T19:11:33.971349+0000","last_unstale":"2024-04-25T19:11:35.470450+0000","last_undegraded":"2024-04-25T19:11:35.470450+0000","last_fullsized":"2024-04-25T19:11:35.470450+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:43.916700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990771+0000","last_change":"2024-04-25T19:11:29.929623+0000","last_active":"2024-04-25T19:11:36.990771+0000","last_peered":"2024-04-25T19:11:36.990771+0000","last_clean":"2024-04-25T19:11:36.990771+0000","last_became_active":"2024-04-25T19:11:29.929306+0000","last_became_peered":"2024-04-25T19:11:29.929306+0000","last_unstale":"2024-04-25T19:11:36.990771+0000","last_undegraded":"2024-04-25T19:11:36.990771+0000","last_fullsized":"2024-04-25T19:11:36.990771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:49.958865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963468+0000","last_change":"2024-04-25T19:11:28.933056+0000","last_active":"2024-04-25T19:11:34.963468+0000","last_peered":"2024-04-25T19:11:34.963468+0000","last_clean":"2024-04-25T19:11:34.963468+0000","last_became_active":"2024-04-25T19:11:28.930233+0000","last_became_peered":"2024-04-25T19:11:28.930233+0000","last_unstale":"2024-04-25T19:11:34.963468+0000","last_undegraded":"2024-04-25T19:11:34.963468+0000","last_fullsized":"2024-04-25T19:11:34.963468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470757+0000","last_change":"2024-04-25T19:11:33.972447+0000","last_active":"2024-04-25T19:11:35.470757+0000","last_peered":"2024-04-25T19:11:35.470757+0000","last_clean":"2024-04-25T19:11:35.470757+0000","last_became_active":"2024-04-25T19:11:33.970645+0000","last_became_peered":"2024-04-25T19:11:33.970645+0000","last_unstale":"2024-04-25T19:11:35.470757+0000","last_undegraded":"2024-04-25T19:11:35.470757+0000","last_fullsized":"2024-04-25T19:11:35.470757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:40.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470256+0000","last_change":"2024-04-25T19:11:29.930967+0000","last_active":"2024-04-25T19:11:35.470256+0000","last_peered":"2024-04-25T19:11:35.470256+0000","last_clean":"2024-04-25T19:11:35.470256+0000","last_became_active":"2024-04-25T19:11:29.930682+0000","last_became_peered":"2024-04-25T19:11:29.930682+0000","last_unstale":"2024-04-25T19:11:35.470256+0000","last_undegraded":"2024-04-25T19:11:35.470256+0000","last_fullsized":"2024-04-25T19:11:35.470256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960304+0000","last_change":"2024-04-25T19:11:28.932205+0000","last_active":"2024-04-25T19:11:34.960304+0000","last_peered":"2024-04-25T19:11:34.960304+0000","last_clean":"2024-04-25T19:11:34.960304+0000","last_became_active":"2024-04-25T19:11:28.931851+0000","last_became_peered":"2024-04-25T19:11:28.931851+0000","last_unstale":"2024-04-25T19:11:34.960304+0000","last_undegraded":"2024-04-25T19:11:34.960304+0000","last_fullsized":"2024-04-25T19:11:34.960304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.471128+0000","last_change":"2024-04-25T19:11:33.971485+0000","last_active":"2024-04-25T19:11:35.471128+0000","last_peered":"2024-04-25T19:11:35.471128+0000","last_clean":"2024-04-25T19:11:35.471128+0000","last_became_active":"2024-04-25T19:11:33.971022+0000","last_became_peered":"2024-04-25T19:11:33.971022+0000","last_unstale":"2024-04-25T19:11:35.471128+0000","last_undegraded":"2024-04-25T19:11:35.471128+0000","last_fullsized":"2024-04-25T19:11:35.471128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:38.690554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878004+0000","last_change":"2024-04-25T19:11:29.936287+0000","last_active":"2024-04-25T19:11:35.878004+0000","last_peered":"2024-04-25T19:11:35.878004+0000","last_clean":"2024-04-25T19:11:35.878004+0000","last_became_active":"2024-04-25T19:11:29.935945+0000","last_became_peered":"2024-04-25T19:11:29.935945+0000","last_unstale":"2024-04-25T19:11:35.878004+0000","last_undegraded":"2024-04-25T19:11:35.878004+0000","last_fullsized":"2024-04-25T19:11:35.878004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:41.846903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170920+0000","last_change":"2024-04-25T19:11:28.940966+0000","last_active":"2024-04-25T19:11:37.170920+0000","last_peered":"2024-04-25T19:11:37.170920+0000","last_clean":"2024-04-25T19:11:37.170920+0000","last_became_active":"2024-04-25T19:11:28.939773+0000","last_became_peered":"2024-04-25T19:11:28.939773+0000","last_unstale":"2024-04-25T19:11:37.170920+0000","last_undegraded":"2024-04-25T19:11:37.170920+0000","last_fullsized":"2024-04-25T19:11:37.170920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878537+0000","last_change":"2024-04-25T19:11:34.113282+0000","last_active":"2024-04-25T19:11:35.878537+0000","last_peered":"2024-04-25T19:11:35.878537+0000","last_clean":"2024-04-25T19:11:35.878537+0000","last_became_active":"2024-04-25T19:11:34.112444+0000","last_became_peered":"2024-04-25T19:11:34.112444+0000","last_unstale":"2024-04-25T19:11:35.878537+0000","last_undegraded":"2024-04-25T19:11:35.878537+0000","last_fullsized":"2024-04-25T19:11:35.878537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:43.702771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170832+0000","last_change":"2024-04-25T19:11:29.949528+0000","last_active":"2024-04-25T19:11:37.170832+0000","last_peered":"2024-04-25T19:11:37.170832+0000","last_clean":"2024-04-25T19:11:37.170832+0000","last_became_active":"2024-04-25T19:11:29.948760+0000","last_became_peered":"2024-04-25T19:11:29.948760+0000","last_unstale":"2024-04-25T19:11:37.170832+0000","last_undegraded":"2024-04-25T19:11:37.170832+0000","last_fullsized":"2024-04-25T19:11:37.170832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963446+0000","last_change":"2024-04-25T19:11:28.928976+0000","last_active":"2024-04-25T19:11:34.963446+0000","last_peered":"2024-04-25T19:11:34.963446+0000","last_clean":"2024-04-25T19:11:34.963446+0000","last_became_active":"2024-04-25T19:11:28.928563+0000","last_became_peered":"2024-04-25T19:11:28.928563+0000","last_unstale":"2024-04-25T19:11:34.963446+0000","last_undegraded":"2024-04-25T19:11:34.963446+0000","last_fullsized":"2024-04-25T19:11:34.963446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":117,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171444+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:11:37.171444+0000","last_peered":"2024-04-25T19:11:37.171444+0000","last_clean":"2024-04-25T19:11:37.171444+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:11:37.171444+0000","last_undegraded":"2024-04-25T19:11:37.171444+0000","last_fullsized":"2024-04-25T19:11:37.171444+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962407+0000","last_change":"2024-04-25T19:11:34.113394+0000","last_active":"2024-04-25T19:11:34.962407+0000","last_peered":"2024-04-25T19:11:34.962407+0000","last_clean":"2024-04-25T19:11:34.962407+0000","last_became_active":"2024-04-25T19:11:34.112950+0000","last_became_peered":"2024-04-25T19:11:34.112950+0000","last_unstale":"2024-04-25T19:11:34.962407+0000","last_undegraded":"2024-04-25T19:11:34.962407+0000","last_fullsized":"2024-04-25T19:11:34.962407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:43.916700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171981+0000","last_change":"2024-04-25T19:11:29.946818+0000","last_active":"2024-04-25T19:11:37.171981+0000","last_peered":"2024-04-25T19:11:37.171981+0000","last_clean":"2024-04-25T19:11:37.171981+0000","last_became_active":"2024-04-25T19:11:29.946517+0000","last_became_peered":"2024-04-25T19:11:29.946517+0000","last_unstale":"2024-04-25T19:11:37.171981+0000","last_undegraded":"2024-04-25T19:11:37.171981+0000","last_fullsized":"2024-04-25T19:11:37.171981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960736+0000","last_change":"2024-04-25T19:11:28.928753+0000","last_active":"2024-04-25T19:11:34.960736+0000","last_peered":"2024-04-25T19:11:34.960736+0000","last_clean":"2024-04-25T19:11:34.960736+0000","last_became_active":"2024-04-25T19:11:28.928388+0000","last_became_peered":"2024-04-25T19:11:28.928388+0000","last_unstale":"2024-04-25T19:11:34.960736+0000","last_undegraded":"2024-04-25T19:11:34.960736+0000","last_fullsized":"2024-04-25T19:11:34.960736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961773+0000","last_change":"2024-04-25T19:11:34.110545+0000","last_active":"2024-04-25T19:11:34.961773+0000","last_peered":"2024-04-25T19:11:34.961773+0000","last_clean":"2024-04-25T19:11:34.961773+0000","last_became_active":"2024-04-25T19:11:34.110024+0000","last_became_peered":"2024-04-25T19:11:34.110024+0000","last_unstale":"2024-04-25T19:11:34.961773+0000","last_undegraded":"2024-04-25T19:11:34.961773+0000","last_fullsized":"2024-04-25T19:11:34.961773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:46.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962527+0000","last_change":"2024-04-25T19:11:29.950030+0000","last_active":"2024-04-25T19:11:34.962527+0000","last_peered":"2024-04-25T19:11:34.962527+0000","last_clean":"2024-04-25T19:11:34.962527+0000","last_became_active":"2024-04-25T19:11:29.949424+0000","last_became_peered":"2024-04-25T19:11:29.949424+0000","last_unstale":"2024-04-25T19:11:34.962527+0000","last_undegraded":"2024-04-25T19:11:34.962527+0000","last_fullsized":"2024-04-25T19:11:34.962527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:48.917939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470828+0000","last_change":"2024-04-25T19:11:28.934208+0000","last_active":"2024-04-25T19:11:35.470828+0000","last_peered":"2024-04-25T19:11:35.470828+0000","last_clean":"2024-04-25T19:11:35.470828+0000","last_became_active":"2024-04-25T19:11:28.933866+0000","last_became_peered":"2024-04-25T19:11:28.933866+0000","last_unstale":"2024-04-25T19:11:35.470828+0000","last_undegraded":"2024-04-25T19:11:35.470828+0000","last_fullsized":"2024-04-25T19:11:35.470828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:30.900062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878372+0000","last_change":"2024-04-25T19:11:33.980318+0000","last_active":"2024-04-25T19:11:35.878372+0000","last_peered":"2024-04-25T19:11:35.878372+0000","last_clean":"2024-04-25T19:11:35.878372+0000","last_became_active":"2024-04-25T19:11:33.979818+0000","last_became_peered":"2024-04-25T19:11:33.979818+0000","last_unstale":"2024-04-25T19:11:35.878372+0000","last_undegraded":"2024-04-25T19:11:35.878372+0000","last_fullsized":"2024-04-25T19:11:35.878372+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:38.690554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172468+0000","last_change":"2024-04-25T19:11:29.940479+0000","last_active":"2024-04-25T19:11:37.172468+0000","last_peered":"2024-04-25T19:11:37.172468+0000","last_clean":"2024-04-25T19:11:37.172468+0000","last_became_active":"2024-04-25T19:11:29.940140+0000","last_became_peered":"2024-04-25T19:11:29.940140+0000","last_unstale":"2024-04-25T19:11:37.172468+0000","last_undegraded":"2024-04-25T19:11:37.172468+0000","last_fullsized":"2024-04-25T19:11:37.172468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:56.761797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960699+0000","last_change":"2024-04-25T19:11:28.933352+0000","last_active":"2024-04-25T19:11:34.960699+0000","last_peered":"2024-04-25T19:11:34.960699+0000","last_clean":"2024-04-25T19:11:34.960699+0000","last_became_active":"2024-04-25T19:11:28.933016+0000","last_became_peered":"2024-04-25T19:11:28.933016+0000","last_unstale":"2024-04-25T19:11:34.960699+0000","last_undegraded":"2024-04-25T19:11:34.960699+0000","last_fullsized":"2024-04-25T19:11:34.960699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961463+0000","last_change":"2024-04-25T19:11:34.110567+0000","last_active":"2024-04-25T19:11:34.961463+0000","last_peered":"2024-04-25T19:11:34.961463+0000","last_clean":"2024-04-25T19:11:34.961463+0000","last_became_active":"2024-04-25T19:11:34.110074+0000","last_became_peered":"2024-04-25T19:11:34.110074+0000","last_unstale":"2024-04-25T19:11:34.961463+0000","last_undegraded":"2024-04-25T19:11:34.961463+0000","last_fullsized":"2024-04-25T19:11:34.961463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:54.738252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172199+0000","last_change":"2024-04-25T19:11:29.939683+0000","last_active":"2024-04-25T19:11:37.172199+0000","last_peered":"2024-04-25T19:11:37.172199+0000","last_clean":"2024-04-25T19:11:37.172199+0000","last_became_active":"2024-04-25T19:11:29.939372+0000","last_became_peered":"2024-04-25T19:11:29.939372+0000","last_unstale":"2024-04-25T19:11:37.172199+0000","last_undegraded":"2024-04-25T19:11:37.172199+0000","last_fullsized":"2024-04-25T19:11:37.172199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:51.175553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989410+0000","last_change":"2024-04-25T19:11:28.928627+0000","last_active":"2024-04-25T19:11:36.989410+0000","last_peered":"2024-04-25T19:11:36.989410+0000","last_clean":"2024-04-25T19:11:36.989410+0000","last_became_active":"2024-04-25T19:11:28.928016+0000","last_became_peered":"2024-04-25T19:11:28.928016+0000","last_unstale":"2024-04-25T19:11:36.989410+0000","last_undegraded":"2024-04-25T19:11:36.989410+0000","last_fullsized":"2024-04-25T19:11:36.989410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:16.833337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989757+0000","last_change":"2024-04-25T19:11:33.971762+0000","last_active":"2024-04-25T19:11:36.989757+0000","last_peered":"2024-04-25T19:11:36.989757+0000","last_clean":"2024-04-25T19:11:36.989757+0000","last_became_active":"2024-04-25T19:11:33.971317+0000","last_became_peered":"2024-04-25T19:11:33.971317+0000","last_unstale":"2024-04-25T19:11:36.989757+0000","last_undegraded":"2024-04-25T19:11:36.989757+0000","last_fullsized":"2024-04-25T19:11:36.989757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:23.748216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171561+0000","last_change":"2024-04-25T19:11:29.938906+0000","last_active":"2024-04-25T19:11:37.171561+0000","last_peered":"2024-04-25T19:11:37.171561+0000","last_clean":"2024-04-25T19:11:37.171561+0000","last_became_active":"2024-04-25T19:11:29.938575+0000","last_became_peered":"2024-04-25T19:11:29.938575+0000","last_unstale":"2024-04-25T19:11:37.171561+0000","last_undegraded":"2024-04-25T19:11:37.171561+0000","last_fullsized":"2024-04-25T19:11:37.171561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960132+0000","last_change":"2024-04-25T19:11:28.931773+0000","last_active":"2024-04-25T19:11:34.960132+0000","last_peered":"2024-04-25T19:11:34.960132+0000","last_clean":"2024-04-25T19:11:34.960132+0000","last_became_active":"2024-04-25T19:11:28.931409+0000","last_became_peered":"2024-04-25T19:11:28.931409+0000","last_unstale":"2024-04-25T19:11:34.960132+0000","last_undegraded":"2024-04-25T19:11:34.960132+0000","last_fullsized":"2024-04-25T19:11:34.960132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:27:30.904833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878361+0000","last_change":"2024-04-25T19:11:33.975482+0000","last_active":"2024-04-25T19:11:35.878361+0000","last_peered":"2024-04-25T19:11:35.878361+0000","last_clean":"2024-04-25T19:11:35.878361+0000","last_became_active":"2024-04-25T19:11:33.975042+0000","last_became_peered":"2024-04-25T19:11:33.975042+0000","last_unstale":"2024-04-25T19:11:35.878361+0000","last_undegraded":"2024-04-25T19:11:35.878361+0000","last_fullsized":"2024-04-25T19:11:35.878361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.795134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961644+0000","last_change":"2024-04-25T19:11:29.943783+0000","last_active":"2024-04-25T19:11:34.961644+0000","last_peered":"2024-04-25T19:11:34.961644+0000","last_clean":"2024-04-25T19:11:34.961644+0000","last_became_active":"2024-04-25T19:11:29.942647+0000","last_became_peered":"2024-04-25T19:11:29.942647+0000","last_unstale":"2024-04-25T19:11:34.961644+0000","last_undegraded":"2024-04-25T19:11:34.961644+0000","last_fullsized":"2024-04-25T19:11:34.961644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961794+0000","last_change":"2024-04-25T19:11:28.934715+0000","last_active":"2024-04-25T19:11:34.961794+0000","last_peered":"2024-04-25T19:11:34.961794+0000","last_clean":"2024-04-25T19:11:34.961794+0000","last_became_active":"2024-04-25T19:11:28.934331+0000","last_became_peered":"2024-04-25T19:11:28.934331+0000","last_unstale":"2024-04-25T19:11:34.961794+0000","last_undegraded":"2024-04-25T19:11:34.961794+0000","last_fullsized":"2024-04-25T19:11:34.961794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171897+0000","last_change":"2024-04-25T19:11:34.115885+0000","last_active":"2024-04-25T19:11:37.171897+0000","last_peered":"2024-04-25T19:11:37.171897+0000","last_clean":"2024-04-25T19:11:37.171897+0000","last_became_active":"2024-04-25T19:11:34.115489+0000","last_became_peered":"2024-04-25T19:11:34.115489+0000","last_unstale":"2024-04-25T19:11:37.171897+0000","last_undegraded":"2024-04-25T19:11:37.171897+0000","last_fullsized":"2024-04-25T19:11:37.171897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:15.088779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171931+0000","last_change":"2024-04-25T19:11:29.937605+0000","last_active":"2024-04-25T19:11:37.171931+0000","last_peered":"2024-04-25T19:11:37.171931+0000","last_clean":"2024-04-25T19:11:37.171931+0000","last_became_active":"2024-04-25T19:11:29.937045+0000","last_became_peered":"2024-04-25T19:11:29.937045+0000","last_unstale":"2024-04-25T19:11:37.171931+0000","last_undegraded":"2024-04-25T19:11:37.171931+0000","last_fullsized":"2024-04-25T19:11:37.171931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:39.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963413+0000","last_change":"2024-04-25T19:11:28.937550+0000","last_active":"2024-04-25T19:11:34.963413+0000","last_peered":"2024-04-25T19:11:34.963413+0000","last_clean":"2024-04-25T19:11:34.963413+0000","last_became_active":"2024-04-25T19:11:28.937224+0000","last_became_peered":"2024-04-25T19:11:28.937224+0000","last_unstale":"2024-04-25T19:11:34.963413+0000","last_undegraded":"2024-04-25T19:11:34.963413+0000","last_fullsized":"2024-04-25T19:11:34.963413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171513+0000","last_change":"2024-04-25T19:11:34.112538+0000","last_active":"2024-04-25T19:11:37.171513+0000","last_peered":"2024-04-25T19:11:37.171513+0000","last_clean":"2024-04-25T19:11:37.171513+0000","last_became_active":"2024-04-25T19:11:34.112105+0000","last_became_peered":"2024-04-25T19:11:34.112105+0000","last_unstale":"2024-04-25T19:11:37.171513+0000","last_undegraded":"2024-04-25T19:11:37.171513+0000","last_fullsized":"2024-04-25T19:11:37.171513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:04.753913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878413+0000","last_change":"2024-04-25T19:11:29.940791+0000","last_active":"2024-04-25T19:11:35.878413+0000","last_peered":"2024-04-25T19:11:35.878413+0000","last_clean":"2024-04-25T19:11:35.878413+0000","last_became_active":"2024-04-25T19:11:29.940494+0000","last_became_peered":"2024-04-25T19:11:29.940494+0000","last_unstale":"2024-04-25T19:11:35.878413+0000","last_undegraded":"2024-04-25T19:11:35.878413+0000","last_fullsized":"2024-04-25T19:11:35.878413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878662+0000","last_change":"2024-04-25T19:11:28.931517+0000","last_active":"2024-04-25T19:11:35.878662+0000","last_peered":"2024-04-25T19:11:35.878662+0000","last_clean":"2024-04-25T19:11:35.878662+0000","last_became_active":"2024-04-25T19:11:28.930964+0000","last_became_peered":"2024-04-25T19:11:28.930964+0000","last_unstale":"2024-04-25T19:11:35.878662+0000","last_undegraded":"2024-04-25T19:11:35.878662+0000","last_fullsized":"2024-04-25T19:11:35.878662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:14.977696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962359+0000","last_change":"2024-04-25T19:11:34.113386+0000","last_active":"2024-04-25T19:11:34.962359+0000","last_peered":"2024-04-25T19:11:34.962359+0000","last_clean":"2024-04-25T19:11:34.962359+0000","last_became_active":"2024-04-25T19:11:34.112820+0000","last_became_peered":"2024-04-25T19:11:34.112820+0000","last_unstale":"2024-04-25T19:11:34.962359+0000","last_undegraded":"2024-04-25T19:11:34.962359+0000","last_fullsized":"2024-04-25T19:11:34.962359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:04.753913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171892+0000","last_change":"2024-04-25T19:11:29.933966+0000","last_active":"2024-04-25T19:11:37.171892+0000","last_peered":"2024-04-25T19:11:37.171892+0000","last_clean":"2024-04-25T19:11:37.171892+0000","last_became_active":"2024-04-25T19:11:29.933549+0000","last_became_peered":"2024-04-25T19:11:29.933549+0000","last_unstale":"2024-04-25T19:11:37.171892+0000","last_undegraded":"2024-04-25T19:11:37.171892+0000","last_fullsized":"2024-04-25T19:11:37.171892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:39.517432+0000","last_change":"2024-04-25T19:11:28.936547+0000","last_active":"2024-04-25T19:11:39.517432+0000","last_peered":"2024-04-25T19:11:39.517432+0000","last_clean":"2024-04-25T19:11:39.517432+0000","last_became_active":"2024-04-25T19:11:28.935816+0000","last_became_peered":"2024-04-25T19:11:28.935816+0000","last_unstale":"2024-04-25T19:11:39.517432+0000","last_undegraded":"2024-04-25T19:11:39.517432+0000","last_fullsized":"2024-04-25T19:11:39.517432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470655+0000","last_change":"2024-04-25T19:11:33.972776+0000","last_active":"2024-04-25T19:11:35.470655+0000","last_peered":"2024-04-25T19:11:35.470655+0000","last_clean":"2024-04-25T19:11:35.470655+0000","last_became_active":"2024-04-25T19:11:33.971757+0000","last_became_peered":"2024-04-25T19:11:33.971757+0000","last_unstale":"2024-04-25T19:11:35.470655+0000","last_undegraded":"2024-04-25T19:11:35.470655+0000","last_fullsized":"2024-04-25T19:11:35.470655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:35.687485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990580+0000","last_change":"2024-04-25T19:11:29.925691+0000","last_active":"2024-04-25T19:11:36.990580+0000","last_peered":"2024-04-25T19:11:36.990580+0000","last_clean":"2024-04-25T19:11:36.990580+0000","last_became_active":"2024-04-25T19:11:29.924990+0000","last_became_peered":"2024-04-25T19:11:29.924990+0000","last_unstale":"2024-04-25T19:11:36.990580+0000","last_undegraded":"2024-04-25T19:11:36.990580+0000","last_fullsized":"2024-04-25T19:11:36.990580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:39.242588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470697+0000","last_change":"2024-04-25T19:11:28.937077+0000","last_active":"2024-04-25T19:11:35.470697+0000","last_peered":"2024-04-25T19:11:35.470697+0000","last_clean":"2024-04-25T19:11:35.470697+0000","last_became_active":"2024-04-25T19:11:28.936181+0000","last_became_peered":"2024-04-25T19:11:28.936181+0000","last_unstale":"2024-04-25T19:11:35.470697+0000","last_undegraded":"2024-04-25T19:11:35.470697+0000","last_fullsized":"2024-04-25T19:11:35.470697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990233+0000","last_change":"2024-04-25T19:11:33.972375+0000","last_active":"2024-04-25T19:11:36.990233+0000","last_peered":"2024-04-25T19:11:36.990233+0000","last_clean":"2024-04-25T19:11:36.990233+0000","last_became_active":"2024-04-25T19:11:33.971792+0000","last_became_peered":"2024-04-25T19:11:33.971792+0000","last_unstale":"2024-04-25T19:11:36.990233+0000","last_undegraded":"2024-04-25T19:11:36.990233+0000","last_fullsized":"2024-04-25T19:11:36.990233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:13.742925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878422+0000","last_change":"2024-04-25T19:11:29.943424+0000","last_active":"2024-04-25T19:11:35.878422+0000","last_peered":"2024-04-25T19:11:35.878422+0000","last_clean":"2024-04-25T19:11:35.878422+0000","last_became_active":"2024-04-25T19:11:29.943096+0000","last_became_peered":"2024-04-25T19:11:29.943096+0000","last_unstale":"2024-04-25T19:11:35.878422+0000","last_undegraded":"2024-04-25T19:11:35.878422+0000","last_fullsized":"2024-04-25T19:11:35.878422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962894+0000","last_change":"2024-04-25T19:11:28.934405+0000","last_active":"2024-04-25T19:11:34.962894+0000","last_peered":"2024-04-25T19:11:34.962894+0000","last_clean":"2024-04-25T19:11:34.962894+0000","last_became_active":"2024-04-25T19:11:28.933743+0000","last_became_peered":"2024-04-25T19:11:28.933743+0000","last_unstale":"2024-04-25T19:11:34.962894+0000","last_undegraded":"2024-04-25T19:11:34.962894+0000","last_fullsized":"2024-04-25T19:11:34.962894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171201+0000","last_change":"2024-04-25T19:11:33.969689+0000","last_active":"2024-04-25T19:11:37.171201+0000","last_peered":"2024-04-25T19:11:37.171201+0000","last_clean":"2024-04-25T19:11:37.171201+0000","last_became_active":"2024-04-25T19:11:33.969257+0000","last_became_peered":"2024-04-25T19:11:33.969257+0000","last_unstale":"2024-04-25T19:11:37.171201+0000","last_undegraded":"2024-04-25T19:11:37.171201+0000","last_fullsized":"2024-04-25T19:11:37.171201+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:57.736550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962757+0000","last_change":"2024-04-25T19:11:29.949944+0000","last_active":"2024-04-25T19:11:34.962757+0000","last_peered":"2024-04-25T19:11:34.962757+0000","last_clean":"2024-04-25T19:11:34.962757+0000","last_became_active":"2024-04-25T19:11:29.949521+0000","last_became_peered":"2024-04-25T19:11:29.949521+0000","last_unstale":"2024-04-25T19:11:34.962757+0000","last_undegraded":"2024-04-25T19:11:34.962757+0000","last_fullsized":"2024-04-25T19:11:34.962757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963008+0000","last_change":"2024-04-25T19:11:28.942291+0000","last_active":"2024-04-25T19:11:34.963008+0000","last_peered":"2024-04-25T19:11:34.963008+0000","last_clean":"2024-04-25T19:11:34.963008+0000","last_became_active":"2024-04-25T19:11:28.941951+0000","last_became_peered":"2024-04-25T19:11:28.941951+0000","last_unstale":"2024-04-25T19:11:34.963008+0000","last_undegraded":"2024-04-25T19:11:34.963008+0000","last_fullsized":"2024-04-25T19:11:34.963008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171184+0000","last_change":"2024-04-25T19:11:34.114829+0000","last_active":"2024-04-25T19:11:37.171184+0000","last_peered":"2024-04-25T19:11:37.171184+0000","last_clean":"2024-04-25T19:11:37.171184+0000","last_became_active":"2024-04-25T19:11:34.114452+0000","last_became_peered":"2024-04-25T19:11:34.114452+0000","last_unstale":"2024-04-25T19:11:37.171184+0000","last_undegraded":"2024-04-25T19:11:37.171184+0000","last_fullsized":"2024-04-25T19:11:37.171184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:54.738252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878339+0000","last_change":"2024-04-25T19:11:34.114794+0000","last_active":"2024-04-25T19:11:35.878339+0000","last_peered":"2024-04-25T19:11:35.878339+0000","last_clean":"2024-04-25T19:11:35.878339+0000","last_became_active":"2024-04-25T19:11:34.114376+0000","last_became_peered":"2024-04-25T19:11:34.114376+0000","last_unstale":"2024-04-25T19:11:35.878339+0000","last_undegraded":"2024-04-25T19:11:35.878339+0000","last_fullsized":"2024-04-25T19:11:35.878339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:40.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470153+0000","last_change":"2024-04-25T19:11:29.935850+0000","last_active":"2024-04-25T19:11:35.470153+0000","last_peered":"2024-04-25T19:11:35.470153+0000","last_clean":"2024-04-25T19:11:35.470153+0000","last_became_active":"2024-04-25T19:11:29.935546+0000","last_became_peered":"2024-04-25T19:11:29.935546+0000","last_unstale":"2024-04-25T19:11:35.470153+0000","last_undegraded":"2024-04-25T19:11:35.470153+0000","last_fullsized":"2024-04-25T19:11:35.470153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961554+0000","last_change":"2024-04-25T19:11:28.928204+0000","last_active":"2024-04-25T19:11:34.961554+0000","last_peered":"2024-04-25T19:11:34.961554+0000","last_clean":"2024-04-25T19:11:34.961554+0000","last_became_active":"2024-04-25T19:11:28.927812+0000","last_became_peered":"2024-04-25T19:11:28.927812+0000","last_unstale":"2024-04-25T19:11:34.961554+0000","last_undegraded":"2024-04-25T19:11:34.961554+0000","last_fullsized":"2024-04-25T19:11:34.961554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961976+0000","last_change":"2024-04-25T19:11:33.971937+0000","last_active":"2024-04-25T19:11:34.961976+0000","last_peered":"2024-04-25T19:11:34.961976+0000","last_clean":"2024-04-25T19:11:34.961976+0000","last_became_active":"2024-04-25T19:11:33.971297+0000","last_became_peered":"2024-04-25T19:11:33.971297+0000","last_unstale":"2024-04-25T19:11:34.961976+0000","last_undegraded":"2024-04-25T19:11:34.961976+0000","last_fullsized":"2024-04-25T19:11:34.961976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:52.951275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171272+0000","last_change":"2024-04-25T19:11:29.936690+0000","last_active":"2024-04-25T19:11:37.171272+0000","last_peered":"2024-04-25T19:11:37.171272+0000","last_clean":"2024-04-25T19:11:37.171272+0000","last_became_active":"2024-04-25T19:11:29.934968+0000","last_became_peered":"2024-04-25T19:11:29.934968+0000","last_unstale":"2024-04-25T19:11:37.171272+0000","last_undegraded":"2024-04-25T19:11:37.171272+0000","last_fullsized":"2024-04-25T19:11:37.171272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170564+0000","last_change":"2024-04-25T19:11:28.936664+0000","last_active":"2024-04-25T19:11:37.170564+0000","last_peered":"2024-04-25T19:11:37.170564+0000","last_clean":"2024-04-25T19:11:37.170564+0000","last_became_active":"2024-04-25T19:11:28.936236+0000","last_became_peered":"2024-04-25T19:11:28.936236+0000","last_unstale":"2024-04-25T19:11:37.170564+0000","last_undegraded":"2024-04-25T19:11:37.170564+0000","last_fullsized":"2024-04-25T19:11:37.170564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962069+0000","last_change":"2024-04-25T19:11:28.934644+0000","last_active":"2024-04-25T19:11:34.962069+0000","last_peered":"2024-04-25T19:11:34.962069+0000","last_clean":"2024-04-25T19:11:34.962069+0000","last_became_active":"2024-04-25T19:11:28.934282+0000","last_became_peered":"2024-04-25T19:11:28.934282+0000","last_unstale":"2024-04-25T19:11:34.962069+0000","last_undegraded":"2024-04-25T19:11:34.962069+0000","last_fullsized":"2024-04-25T19:11:34.962069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878411+0000","last_change":"2024-04-25T19:11:29.934419+0000","last_active":"2024-04-25T19:11:35.878411+0000","last_peered":"2024-04-25T19:11:35.878411+0000","last_clean":"2024-04-25T19:11:35.878411+0000","last_became_active":"2024-04-25T19:11:29.934119+0000","last_became_peered":"2024-04-25T19:11:29.934119+0000","last_unstale":"2024-04-25T19:11:35.878411+0000","last_undegraded":"2024-04-25T19:11:35.878411+0000","last_fullsized":"2024-04-25T19:11:35.878411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960537+0000","last_change":"2024-04-25T19:11:33.969976+0000","last_active":"2024-04-25T19:11:34.960537+0000","last_peered":"2024-04-25T19:11:34.960537+0000","last_clean":"2024-04-25T19:11:34.960537+0000","last_became_active":"2024-04-25T19:11:33.969493+0000","last_became_peered":"2024-04-25T19:11:33.969493+0000","last_unstale":"2024-04-25T19:11:34.960537+0000","last_undegraded":"2024-04-25T19:11:34.960537+0000","last_fullsized":"2024-04-25T19:11:34.960537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:45.880240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171502+0000","last_change":"2024-04-25T19:11:28.933487+0000","last_active":"2024-04-25T19:11:37.171502+0000","last_peered":"2024-04-25T19:11:37.171502+0000","last_clean":"2024-04-25T19:11:37.171502+0000","last_became_active":"2024-04-25T19:11:28.933112+0000","last_became_peered":"2024-04-25T19:11:28.933112+0000","last_unstale":"2024-04-25T19:11:37.171502+0000","last_undegraded":"2024-04-25T19:11:37.171502+0000","last_fullsized":"2024-04-25T19:11:37.171502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:44.780178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990049+0000","last_change":"2024-04-25T19:11:29.927210+0000","last_active":"2024-04-25T19:11:36.990049+0000","last_peered":"2024-04-25T19:11:36.990049+0000","last_clean":"2024-04-25T19:11:36.990049+0000","last_became_active":"2024-04-25T19:11:29.926143+0000","last_became_peered":"2024-04-25T19:11:29.926143+0000","last_unstale":"2024-04-25T19:11:36.990049+0000","last_undegraded":"2024-04-25T19:11:36.990049+0000","last_fullsized":"2024-04-25T19:11:36.990049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:31.906041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171522+0000","last_change":"2024-04-25T19:11:33.974927+0000","last_active":"2024-04-25T19:11:37.171522+0000","last_peered":"2024-04-25T19:11:37.171522+0000","last_clean":"2024-04-25T19:11:37.171522+0000","last_became_active":"2024-04-25T19:11:33.974526+0000","last_became_peered":"2024-04-25T19:11:33.974526+0000","last_unstale":"2024-04-25T19:11:37.171522+0000","last_undegraded":"2024-04-25T19:11:37.171522+0000","last_fullsized":"2024-04-25T19:11:37.171522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:32.367757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961363+0000","last_change":"2024-04-25T19:11:29.943955+0000","last_active":"2024-04-25T19:11:34.961363+0000","last_peered":"2024-04-25T19:11:34.961363+0000","last_clean":"2024-04-25T19:11:34.961363+0000","last_became_active":"2024-04-25T19:11:29.943620+0000","last_became_peered":"2024-04-25T19:11:29.943620+0000","last_unstale":"2024-04-25T19:11:34.961363+0000","last_undegraded":"2024-04-25T19:11:34.961363+0000","last_fullsized":"2024-04-25T19:11:34.961363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960143+0000","last_change":"2024-04-25T19:11:29.944224+0000","last_active":"2024-04-25T19:11:34.960143+0000","last_peered":"2024-04-25T19:11:34.960143+0000","last_clean":"2024-04-25T19:11:34.960143+0000","last_became_active":"2024-04-25T19:11:29.943851+0000","last_became_peered":"2024-04-25T19:11:29.943851+0000","last_unstale":"2024-04-25T19:11:34.960143+0000","last_undegraded":"2024-04-25T19:11:34.960143+0000","last_fullsized":"2024-04-25T19:11:34.960143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:31.906041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963271+0000","last_change":"2024-04-25T19:11:29.948559+0000","last_active":"2024-04-25T19:11:34.963271+0000","last_peered":"2024-04-25T19:11:34.963271+0000","last_clean":"2024-04-25T19:11:34.963271+0000","last_became_active":"2024-04-25T19:11:29.948261+0000","last_became_peered":"2024-04-25T19:11:29.948261+0000","last_unstale":"2024-04-25T19:11:34.963271+0000","last_undegraded":"2024-04-25T19:11:34.963271+0000","last_fullsized":"2024-04-25T19:11:34.963271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878297+0000","last_change":"2024-04-25T19:11:29.941924+0000","last_active":"2024-04-25T19:11:35.878297+0000","last_peered":"2024-04-25T19:11:35.878297+0000","last_clean":"2024-04-25T19:11:35.878297+0000","last_became_active":"2024-04-25T19:11:29.939317+0000","last_became_peered":"2024-04-25T19:11:29.939317+0000","last_unstale":"2024-04-25T19:11:35.878297+0000","last_undegraded":"2024-04-25T19:11:35.878297+0000","last_fullsized":"2024-04-25T19:11:35.878297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171676+0000","last_change":"2024-04-25T19:11:29.937716+0000","last_active":"2024-04-25T19:11:37.171676+0000","last_peered":"2024-04-25T19:11:37.171676+0000","last_clean":"2024-04-25T19:11:37.171676+0000","last_became_active":"2024-04-25T19:11:29.936902+0000","last_became_peered":"2024-04-25T19:11:29.936902+0000","last_unstale":"2024-04-25T19:11:37.171676+0000","last_undegraded":"2024-04-25T19:11:37.171676+0000","last_fullsized":"2024-04-25T19:11:37.171676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470475+0000","last_change":"2024-04-25T19:11:29.933948+0000","last_active":"2024-04-25T19:11:35.470475+0000","last_peered":"2024-04-25T19:11:35.470475+0000","last_clean":"2024-04-25T19:11:35.470475+0000","last_became_active":"2024-04-25T19:11:29.933609+0000","last_became_peered":"2024-04-25T19:11:29.933609+0000","last_unstale":"2024-04-25T19:11:35.470475+0000","last_undegraded":"2024-04-25T19:11:35.470475+0000","last_fullsized":"2024-04-25T19:11:35.470475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963143+0000","last_change":"2024-04-25T19:11:29.949696+0000","last_active":"2024-04-25T19:11:34.963143+0000","last_peered":"2024-04-25T19:11:34.963143+0000","last_clean":"2024-04-25T19:11:34.963143+0000","last_became_active":"2024-04-25T19:11:29.949244+0000","last_became_peered":"2024-04-25T19:11:29.949244+0000","last_unstale":"2024-04-25T19:11:34.963143+0000","last_undegraded":"2024-04-25T19:11:34.963143+0000","last_fullsized":"2024-04-25T19:11:34.963143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171872+0000","last_change":"2024-04-25T19:11:29.936507+0000","last_active":"2024-04-25T19:11:37.171872+0000","last_peered":"2024-04-25T19:11:37.171872+0000","last_clean":"2024-04-25T19:11:37.171872+0000","last_became_active":"2024-04-25T19:11:29.936194+0000","last_became_peered":"2024-04-25T19:11:29.936194+0000","last_unstale":"2024-04-25T19:11:37.171872+0000","last_undegraded":"2024-04-25T19:11:37.171872+0000","last_fullsized":"2024-04-25T19:11:37.171872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961467+0000","last_change":"2024-04-25T19:11:29.949296+0000","last_active":"2024-04-25T19:11:34.961467+0000","last_peered":"2024-04-25T19:11:34.961467+0000","last_clean":"2024-04-25T19:11:34.961467+0000","last_became_active":"2024-04-25T19:11:29.948939+0000","last_became_peered":"2024-04-25T19:11:29.948939+0000","last_unstale":"2024-04-25T19:11:34.961467+0000","last_undegraded":"2024-04-25T19:11:34.961467+0000","last_fullsized":"2024-04-25T19:11:34.961467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:39.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171201+0000","last_change":"2024-04-25T19:11:29.937245+0000","last_active":"2024-04-25T19:11:37.171201+0000","last_peered":"2024-04-25T19:11:37.171201+0000","last_clean":"2024-04-25T19:11:37.171201+0000","last_became_active":"2024-04-25T19:11:29.936924+0000","last_became_peered":"2024-04-25T19:11:29.936924+0000","last_unstale":"2024-04-25T19:11:37.171201+0000","last_undegraded":"2024-04-25T19:11:37.171201+0000","last_fullsized":"2024-04-25T19:11:37.171201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963148+0000","last_change":"2024-04-25T19:11:29.939024+0000","last_active":"2024-04-25T19:11:34.963148+0000","last_peered":"2024-04-25T19:11:34.963148+0000","last_clean":"2024-04-25T19:11:34.963148+0000","last_became_active":"2024-04-25T19:11:29.938702+0000","last_became_peered":"2024-04-25T19:11:29.938702+0000","last_unstale":"2024-04-25T19:11:34.963148+0000","last_undegraded":"2024-04-25T19:11:34.963148+0000","last_fullsized":"2024-04-25T19:11:34.963148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878155+0000","last_change":"2024-04-25T19:11:29.942883+0000","last_active":"2024-04-25T19:11:35.878155+0000","last_peered":"2024-04-25T19:11:35.878155+0000","last_clean":"2024-04-25T19:11:35.878155+0000","last_became_active":"2024-04-25T19:11:29.942371+0000","last_became_peered":"2024-04-25T19:11:29.942371+0000","last_unstale":"2024-04-25T19:11:35.878155+0000","last_undegraded":"2024-04-25T19:11:35.878155+0000","last_fullsized":"2024-04-25T19:11:35.878155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962242+0000","last_change":"2024-04-25T19:11:29.940287+0000","last_active":"2024-04-25T19:11:34.962242+0000","last_peered":"2024-04-25T19:11:34.962242+0000","last_clean":"2024-04-25T19:11:34.962242+0000","last_became_active":"2024-04-25T19:11:29.939973+0000","last_became_peered":"2024-04-25T19:11:29.939973+0000","last_unstale":"2024-04-25T19:11:34.962242+0000","last_undegraded":"2024-04-25T19:11:34.962242+0000","last_fullsized":"2024-04-25T19:11:34.962242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172189+0000","last_change":"2024-04-25T19:11:29.932331+0000","last_active":"2024-04-25T19:11:37.172189+0000","last_peered":"2024-04-25T19:11:37.172189+0000","last_clean":"2024-04-25T19:11:37.172189+0000","last_became_active":"2024-04-25T19:11:29.932030+0000","last_became_peered":"2024-04-25T19:11:29.932030+0000","last_unstale":"2024-04-25T19:11:37.172189+0000","last_undegraded":"2024-04-25T19:11:37.172189+0000","last_fullsized":"2024-04-25T19:11:37.172189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:48.917939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960501+0000","last_change":"2024-04-25T19:11:29.930099+0000","last_active":"2024-04-25T19:11:34.960501+0000","last_peered":"2024-04-25T19:11:34.960501+0000","last_clean":"2024-04-25T19:11:34.960501+0000","last_became_active":"2024-04-25T19:11:29.929729+0000","last_became_peered":"2024-04-25T19:11:29.929729+0000","last_unstale":"2024-04-25T19:11:34.960501+0000","last_undegraded":"2024-04-25T19:11:34.960501+0000","last_fullsized":"2024-04-25T19:11:34.960501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:39.242588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963790+0000","last_change":"2024-04-25T19:11:29.940716+0000","last_active":"2024-04-25T19:11:34.963790+0000","last_peered":"2024-04-25T19:11:34.963790+0000","last_clean":"2024-04-25T19:11:34.963790+0000","last_became_active":"2024-04-25T19:11:29.940412+0000","last_became_peered":"2024-04-25T19:11:29.940412+0000","last_unstale":"2024-04-25T19:11:34.963790+0000","last_undegraded":"2024-04-25T19:11:34.963790+0000","last_fullsized":"2024-04-25T19:11:34.963790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:35.993870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961354+0000","last_change":"2024-04-25T19:11:29.945896+0000","last_active":"2024-04-25T19:11:34.961354+0000","last_peered":"2024-04-25T19:11:34.961354+0000","last_clean":"2024-04-25T19:11:34.961354+0000","last_became_active":"2024-04-25T19:11:29.945590+0000","last_became_peered":"2024-04-25T19:11:29.945590+0000","last_unstale":"2024-04-25T19:11:34.961354+0000","last_undegraded":"2024-04-25T19:11:34.961354+0000","last_fullsized":"2024-04-25T19:11:34.961354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:42.722556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470247+0000","last_change":"2024-04-25T19:11:29.939162+0000","last_active":"2024-04-25T19:11:35.470247+0000","last_peered":"2024-04-25T19:11:35.470247+0000","last_clean":"2024-04-25T19:11:35.470247+0000","last_became_active":"2024-04-25T19:11:29.938821+0000","last_became_peered":"2024-04-25T19:11:29.938821+0000","last_unstale":"2024-04-25T19:11:35.470247+0000","last_undegraded":"2024-04-25T19:11:35.470247+0000","last_fullsized":"2024-04-25T19:11:35.470247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878300+0000","last_change":"2024-04-25T19:11:29.938034+0000","last_active":"2024-04-25T19:11:35.878300+0000","last_peered":"2024-04-25T19:11:35.878300+0000","last_clean":"2024-04-25T19:11:35.878300+0000","last_became_active":"2024-04-25T19:11:29.936771+0000","last_became_peered":"2024-04-25T19:11:29.936771+0000","last_unstale":"2024-04-25T19:11:35.878300+0000","last_undegraded":"2024-04-25T19:11:35.878300+0000","last_fullsized":"2024-04-25T19:11:35.878300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:23.833133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171095+0000","last_change":"2024-04-25T19:11:29.944382+0000","last_active":"2024-04-25T19:11:37.171095+0000","last_peered":"2024-04-25T19:11:37.171095+0000","last_clean":"2024-04-25T19:11:37.171095+0000","last_became_active":"2024-04-25T19:11:29.938702+0000","last_became_peered":"2024-04-25T19:11:29.938702+0000","last_unstale":"2024-04-25T19:11:37.171095+0000","last_undegraded":"2024-04-25T19:11:37.171095+0000","last_fullsized":"2024-04-25T19:11:37.171095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990488+0000","last_change":"2024-04-25T19:11:29.934918+0000","last_active":"2024-04-25T19:11:36.990488+0000","last_peered":"2024-04-25T19:11:36.990488+0000","last_clean":"2024-04-25T19:11:36.990488+0000","last_became_active":"2024-04-25T19:11:29.934600+0000","last_became_peered":"2024-04-25T19:11:29.934600+0000","last_unstale":"2024-04-25T19:11:36.990488+0000","last_undegraded":"2024-04-25T19:11:36.990488+0000","last_fullsized":"2024-04-25T19:11:36.990488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:15.027554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962499+0000","last_change":"2024-04-25T19:11:29.944854+0000","last_active":"2024-04-25T19:11:34.962499+0000","last_peered":"2024-04-25T19:11:34.962499+0000","last_clean":"2024-04-25T19:11:34.962499+0000","last_became_active":"2024-04-25T19:11:29.944528+0000","last_became_peered":"2024-04-25T19:11:29.944528+0000","last_unstale":"2024-04-25T19:11:34.962499+0000","last_undegraded":"2024-04-25T19:11:34.962499+0000","last_fullsized":"2024-04-25T19:11:34.962499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963024+0000","last_change":"2024-04-25T19:11:29.944061+0000","last_active":"2024-04-25T19:11:34.963024+0000","last_peered":"2024-04-25T19:11:34.963024+0000","last_clean":"2024-04-25T19:11:34.963024+0000","last_became_active":"2024-04-25T19:11:29.943745+0000","last_became_peered":"2024-04-25T19:11:29.943745+0000","last_unstale":"2024-04-25T19:11:34.963024+0000","last_undegraded":"2024-04-25T19:11:34.963024+0000","last_fullsized":"2024-04-25T19:11:34.963024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172433+0000","last_change":"2024-04-25T19:11:29.944585+0000","last_active":"2024-04-25T19:11:37.172433+0000","last_peered":"2024-04-25T19:11:37.172433+0000","last_clean":"2024-04-25T19:11:37.172433+0000","last_became_active":"2024-04-25T19:11:29.944112+0000","last_became_peered":"2024-04-25T19:11:29.944112+0000","last_unstale":"2024-04-25T19:11:37.172433+0000","last_undegraded":"2024-04-25T19:11:37.172433+0000","last_fullsized":"2024-04-25T19:11:37.172433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:31.654149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170915+0000","last_change":"2024-04-25T19:11:29.949991+0000","last_active":"2024-04-25T19:11:37.170915+0000","last_peered":"2024-04-25T19:11:37.170915+0000","last_clean":"2024-04-25T19:11:37.170915+0000","last_became_active":"2024-04-25T19:11:29.948567+0000","last_became_peered":"2024-04-25T19:11:29.948567+0000","last_unstale":"2024-04-25T19:11:37.170915+0000","last_undegraded":"2024-04-25T19:11:37.170915+0000","last_fullsized":"2024-04-25T19:11:37.170915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171414+0000","last_change":"2024-04-25T19:11:29.930321+0000","last_active":"2024-04-25T19:11:37.171414+0000","last_peered":"2024-04-25T19:11:37.171414+0000","last_clean":"2024-04-25T19:11:37.171414+0000","last_became_active":"2024-04-25T19:11:29.929930+0000","last_became_peered":"2024-04-25T19:11:29.929930+0000","last_unstale":"2024-04-25T19:11:37.171414+0000","last_undegraded":"2024-04-25T19:11:37.171414+0000","last_fullsized":"2024-04-25T19:11:37.171414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963030+0000","last_change":"2024-04-25T19:11:29.944694+0000","last_active":"2024-04-25T19:11:34.963030+0000","last_peered":"2024-04-25T19:11:34.963030+0000","last_clean":"2024-04-25T19:11:34.963030+0000","last_became_active":"2024-04-25T19:11:29.944055+0000","last_became_peered":"2024-04-25T19:11:29.944055+0000","last_unstale":"2024-04-25T19:11:34.963030+0000","last_undegraded":"2024-04-25T19:11:34.963030+0000","last_fullsized":"2024-04-25T19:11:34.963030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171775+0000","last_change":"2024-04-25T19:11:29.931815+0000","last_active":"2024-04-25T19:11:37.171775+0000","last_peered":"2024-04-25T19:11:37.171775+0000","last_clean":"2024-04-25T19:11:37.171775+0000","last_became_active":"2024-04-25T19:11:29.931523+0000","last_became_peered":"2024-04-25T19:11:29.931523+0000","last_unstale":"2024-04-25T19:11:37.171775+0000","last_undegraded":"2024-04-25T19:11:37.171775+0000","last_fullsized":"2024-04-25T19:11:37.171775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:42.722556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962270+0000","last_change":"2024-04-25T19:11:29.951038+0000","last_active":"2024-04-25T19:11:34.962270+0000","last_peered":"2024-04-25T19:11:34.962270+0000","last_clean":"2024-04-25T19:11:34.962270+0000","last_became_active":"2024-04-25T19:11:29.950615+0000","last_became_peered":"2024-04-25T19:11:29.950615+0000","last_unstale":"2024-04-25T19:11:34.962270+0000","last_undegraded":"2024-04-25T19:11:34.962270+0000","last_fullsized":"2024-04-25T19:11:34.962270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:31.654149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878604+0000","last_change":"2024-04-25T19:11:29.932658+0000","last_active":"2024-04-25T19:11:35.878604+0000","last_peered":"2024-04-25T19:11:35.878604+0000","last_clean":"2024-04-25T19:11:35.878604+0000","last_became_active":"2024-04-25T19:11:29.932270+0000","last_became_peered":"2024-04-25T19:11:29.932270+0000","last_unstale":"2024-04-25T19:11:35.878604+0000","last_undegraded":"2024-04-25T19:11:35.878604+0000","last_fullsized":"2024-04-25T19:11:35.878604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470928+0000","last_change":"2024-04-25T19:11:29.929765+0000","last_active":"2024-04-25T19:11:35.470928+0000","last_peered":"2024-04-25T19:11:35.470928+0000","last_clean":"2024-04-25T19:11:35.470928+0000","last_became_active":"2024-04-25T19:11:29.929452+0000","last_became_peered":"2024-04-25T19:11:29.929452+0000","last_unstale":"2024-04-25T19:11:35.470928+0000","last_undegraded":"2024-04-25T19:11:35.470928+0000","last_fullsized":"2024-04-25T19:11:35.470928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172357+0000","last_change":"2024-04-25T19:11:29.936916+0000","last_active":"2024-04-25T19:11:37.172357+0000","last_peered":"2024-04-25T19:11:37.172357+0000","last_clean":"2024-04-25T19:11:37.172357+0000","last_became_active":"2024-04-25T19:11:29.935057+0000","last_became_peered":"2024-04-25T19:11:29.935057+0000","last_unstale":"2024-04-25T19:11:37.172357+0000","last_undegraded":"2024-04-25T19:11:37.172357+0000","last_fullsized":"2024-04-25T19:11:37.172357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:35.993870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:11:56.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:11:56.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.178+0000 7fbbd0492640 1 Processor -- start 2024-04-25T19:11:56.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.178+0000 7fbbd0492640 1 -- start start 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.179+0000 7fbbd0492640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbbc8167850 0x7fbbc8169c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.179+0000 7fbbd0492640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbc80d1ce0 con 0x7fbbc809cf00 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.179+0000 7fbbd0492640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbc80cf8d0 con 0x7fbbc809caf0 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.179+0000 7fbbd0492640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fbbc80b8fa0 con 0x7fbbc8167850 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.179+0000 7fbbcda2d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbbc809cf00 0x7fbbc80b71d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59016/0 (socket says 172.21.15.28:59016) 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.179+0000 7fbbcda2d640 1 -- 172.21.15.28:0/1043167564 learned_addr learned my addr 172.21.15.28:0/1043167564 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcea2f640 1 --2- 172.21.15.28:0/1043167564 >> v2:172.21.15.53:3300/0 conn(0x7fbbc8167850 0x7fbbc8169c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3027275516 0 0) 0x7fbbc80d1ce0 con 0x7fbbc809cf00 2024-04-25T19:11:56.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbb4003520 con 0x7fbbc809cf00 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085528769 0 0) 0x7fbbc80cf8d0 con 0x7fbbc809caf0 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbbc80d1ce0 con 0x7fbbc809caf0 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2093624262 0 0) 0x7fbbb4003520 con 0x7fbbc809cf00 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 >> v2:172.21.15.53:3300/0 conn(0x7fbbc8167850 msgr2=0x7fbbc8169c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 --2- 172.21.15.28:0/1043167564 >> v2:172.21.15.53:3300/0 conn(0x7fbbc8167850 0x7fbbc8169c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 >> v1:172.21.15.53:6789/0 conn(0x7fbbc809caf0 legacy=0x7fbbc80d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc80cfbb0 con 0x7fbbc809cf00 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fbbb8001690 con 0x7fbbc809cf00 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 >> v1:172.21.15.28:6789/0 conn(0x7fbbc809cf00 legacy=0x7fbbc80b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbb4003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbb4004800 0x7fbbb4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbb4005260 0x7fbbb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc80b8fa0 con 0x7fbbb4003b40 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb400c1c0 con 0x7fbbb4004800 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb400c3c0 con 0x7fbbb4005260 2024-04-25T19:11:56.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbce22e640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbb4003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcda2d640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbb4004800 0x7fbbb4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.180+0000 7fbbcea2f640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbb4005260 0x7fbbb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbce22e640 1 -- 172.21.15.28:0/1043167564 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbb4005260 msgr2=0x7fbbb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbce22e640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbb4005260 0x7fbbb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbce22e640 1 -- 172.21.15.28:0/1043167564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbb4004800 msgr2=0x7fbbb4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbce22e640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbb4004800 0x7fbbb4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbcea2f640 1 -- 172.21.15.28:0/1043167564 reap_dead start 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbce22e640 1 -- 172.21.15.28:0/1043167564 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb4003520 con 0x7fbbb4003b40 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbce22e640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbb4003f90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80d1470 tx=0x7fbbb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead35b4e88d55333 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40014f0 con 0x7fbbb4003b40 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc40052d0 con 0x7fbbb4003b40 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.181+0000 7fbbcd22c640 1 -- 172.21.15.28:0/1043167564 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc402abe0 con 0x7fbbb4003b40 2024-04-25T19:11:56.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.182+0000 7fbbd0492640 1 -- 172.21.15.28:0/1043167564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 msgr2=0x7fbbb4003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.182+0000 7fbbd0492640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbb4003f90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80d1470 tx=0x7fbbb4005830 comp rx=0 tx=0).stop 2024-04-25T19:11:56.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.182+0000 7fbbd0492640 1 -- 172.21.15.28:0/1043167564 shutdown_connections 2024-04-25T19:11:56.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.182+0000 7fbbd0492640 1 --2- 172.21.15.28:0/1043167564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbb4003f90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.182+0000 7fbbd0492640 1 -- 172.21.15.28:0/1043167564 >> 172.21.15.28:0/1043167564 conn(0x7fbbc80d03f0 msgr2=0x7fbbc80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:56.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.182+0000 7fbbd0492640 1 -- 172.21.15.28:0/1043167564 shutdown_connections 2024-04-25T19:11:56.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.182+0000 7fbbd0492640 1 -- 172.21.15.28:0/1043167564 wait complete. 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.184+0000 7fbbd0492640 1 Processor -- start 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.184+0000 7fbbd0492640 1 -- start start 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbd0492640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbc806f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbd0492640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc80cc590 0x7fbbc8156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbd0492640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc80ce6e0 0x7fbbc80ceaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbd0492640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc80d4790 con 0x7fbbb4003b40 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbd0492640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbbc80b9430 con 0x7fbbc80cc590 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbd0492640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbbc80d2b10 con 0x7fbbc80ce6e0 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbce22e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbc806f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbce22e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbc806f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37392/0 (socket says 172.21.15.28:37392) 2024-04-25T19:11:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.185+0000 7fbbce22e640 1 -- 172.21.15.28:0/2121648134 learned_addr learned my addr 172.21.15.28:0/2121648134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbce22e640 1 -- 172.21.15.28:0/2121648134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc80ce6e0 msgr2=0x7fbbc80ceaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbcda2d640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc80cc590 0x7fbbc8156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbce22e640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc80ce6e0 0x7fbbc80ceaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbce22e640 1 -- 172.21.15.28:0/2121648134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc80cc590 msgr2=0x7fbbc8156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbce22e640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc80cc590 0x7fbbc8156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbce22e640 1 -- 172.21.15.28:0/2121648134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbc80cf030 con 0x7fbbb4003b40 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbce22e640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbc806f330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4001990 tx=0x7fbbc40012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbbeffd640 1 -- 172.21.15.28:0/2121648134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4037040 con 0x7fbbb4003b40 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbbeffd640 1 -- 172.21.15.28:0/2121648134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbc402be50 con 0x7fbbb4003b40 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbc80ccda0 con 0x7fbbb4003b40 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbbeffd640 1 -- 172.21.15.28:0/2121648134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40014f0 con 0x7fbbb4003b40 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.186+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbc81650c0 con 0x7fbbb4003b40 2024-04-25T19:11:56.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.187+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbc8050740 con 0x7fbbb4003b40 2024-04-25T19:11:56.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.188+0000 7fbbbeffd640 1 -- 172.21.15.28:0/2121648134 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbbc4044020 con 0x7fbbb4003b40 2024-04-25T19:11:56.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.188+0000 7fbbbeffd640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbac0744e0 0x7fbbac0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.188+0000 7fbbbeffd640 1 -- 172.21.15.28:0/2121648134 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbbc40b50c0 con 0x7fbbb4003b40 2024-04-25T19:11:56.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.189+0000 7fbbcda2d640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbac0744e0 0x7fbbac0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.189+0000 7fbbcda2d640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbac0744e0 0x7fbbac0769f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbbc81f9540 tx=0x7fbbb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.193+0000 7fbbbeffd640 1 -- 172.21.15.28:0/2121648134 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbbc404b020 con 0x7fbbb4003b40 2024-04-25T19:11:56.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.377+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbbc80cd2c0 con 0x7fbbac0744e0 2024-04-25T19:11:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.385+0000 7fbbbeffd640 1 -- 172.21.15.28:0/2121648134 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332522 (secure 0 0 0) 0x7fbbb8004de0 con 0x7fbbac0744e0 2024-04-25T19:11:56.392 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:56.393 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:11:56.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbac0744e0 msgr2=0x7fbbac0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbac0744e0 0x7fbbac0769f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbbc81f9540 tx=0x7fbbb8009040 comp rx=0 tx=0).stop 2024-04-25T19:11:56.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 msgr2=0x7fbbc806f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbc806f330 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4001990 tx=0x7fbbc40012d0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 shutdown_connections 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbbac0744e0 0x7fbbac0769f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbbc80ce6e0 0x7fbbc80ceaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc80cc590 0x7fbbc8156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 --2- 172.21.15.28:0/2121648134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbb4003b40 0x7fbbc806f330 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 >> 172.21.15.28:0/2121648134 conn(0x7fbbc80d03f0 msgr2=0x7fbbc8071e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 shutdown_connections 2024-04-25T19:11:56.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.389+0000 7fbbd0492640 1 -- 172.21.15.28:0/2121648134 wait complete. 2024-04-25T19:11:56.402 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-25T19:11:54.842514+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252656,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754722064,"statfs":{"total":773094113280,"available":772835393536,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003591"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878592+0000","last_change":"2024-04-25T19:11:28.938792+0000","last_active":"2024-04-25T19:11:35.878592+0000","last_peered":"2024-04-25T19:11:35.878592+0000","last_clean":"2024-04-25T19:11:35.878592+0000","last_became_active":"2024-04-25T19:11:28.938417+0000","last_became_peered":"2024-04-25T19:11:28.938417+0000","last_unstale":"2024-04-25T19:11:35.878592+0000","last_undegraded":"2024-04-25T19:11:35.878592+0000","last_fullsized":"2024-04-25T19:11:35.878592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:44.780178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171775+0000","last_change":"2024-04-25T19:11:29.937823+0000","last_active":"2024-04-25T19:11:37.171775+0000","last_peered":"2024-04-25T19:11:37.171775+0000","last_clean":"2024-04-25T19:11:37.171775+0000","last_became_active":"2024-04-25T19:11:29.937501+0000","last_became_peered":"2024-04-25T19:11:29.937501+0000","last_unstale":"2024-04-25T19:11:37.171775+0000","last_undegraded":"2024-04-25T19:11:37.171775+0000","last_fullsized":"2024-04-25T19:11:37.171775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170976+0000","last_change":"2024-04-25T19:11:34.113864+0000","last_active":"2024-04-25T19:11:37.170976+0000","last_peered":"2024-04-25T19:11:37.170976+0000","last_clean":"2024-04-25T19:11:37.170976+0000","last_became_active":"2024-04-25T19:11:34.113449+0000","last_became_peered":"2024-04-25T19:11:34.113449+0000","last_unstale":"2024-04-25T19:11:37.170976+0000","last_undegraded":"2024-04-25T19:11:37.170976+0000","last_fullsized":"2024-04-25T19:11:37.170976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:46.247905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170712+0000","last_change":"2024-04-25T19:11:28.934472+0000","last_active":"2024-04-25T19:11:37.170712+0000","last_peered":"2024-04-25T19:11:37.170712+0000","last_clean":"2024-04-25T19:11:37.170712+0000","last_became_active":"2024-04-25T19:11:28.934016+0000","last_became_peered":"2024-04-25T19:11:28.934016+0000","last_unstale":"2024-04-25T19:11:37.170712+0000","last_undegraded":"2024-04-25T19:11:37.170712+0000","last_fullsized":"2024-04-25T19:11:37.170712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:27:30.904833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878257+0000","last_change":"2024-04-25T19:11:29.944065+0000","last_active":"2024-04-25T19:11:35.878257+0000","last_peered":"2024-04-25T19:11:35.878257+0000","last_clean":"2024-04-25T19:11:35.878257+0000","last_became_active":"2024-04-25T19:11:29.943773+0000","last_became_peered":"2024-04-25T19:11:29.943773+0000","last_unstale":"2024-04-25T19:11:35.878257+0000","last_undegraded":"2024-04-25T19:11:35.878257+0000","last_fullsized":"2024-04-25T19:11:35.878257+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:16:15.552217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989732+0000","last_change":"2024-04-25T19:11:33.975423+0000","last_active":"2024-04-25T19:11:36.989732+0000","last_peered":"2024-04-25T19:11:36.989732+0000","last_clean":"2024-04-25T19:11:36.989732+0000","last_became_active":"2024-04-25T19:11:33.974878+0000","last_became_peered":"2024-04-25T19:11:33.974878+0000","last_unstale":"2024-04-25T19:11:36.989732+0000","last_undegraded":"2024-04-25T19:11:36.989732+0000","last_fullsized":"2024-04-25T19:11:36.989732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:23.927981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170963+0000","last_change":"2024-04-25T19:11:28.937102+0000","last_active":"2024-04-25T19:11:37.170963+0000","last_peered":"2024-04-25T19:11:37.170963+0000","last_clean":"2024-04-25T19:11:37.170963+0000","last_became_active":"2024-04-25T19:11:28.936699+0000","last_became_peered":"2024-04-25T19:11:28.936699+0000","last_unstale":"2024-04-25T19:11:37.170963+0000","last_undegraded":"2024-04-25T19:11:37.170963+0000","last_fullsized":"2024-04-25T19:11:37.170963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:38.236609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962397+0000","last_change":"2024-04-25T19:11:29.947706+0000","last_active":"2024-04-25T19:11:34.962397+0000","last_peered":"2024-04-25T19:11:34.962397+0000","last_clean":"2024-04-25T19:11:34.962397+0000","last_became_active":"2024-04-25T19:11:29.945963+0000","last_became_peered":"2024-04-25T19:11:29.945963+0000","last_unstale":"2024-04-25T19:11:34.962397+0000","last_undegraded":"2024-04-25T19:11:34.962397+0000","last_fullsized":"2024-04-25T19:11:34.962397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:51.175553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990390+0000","last_change":"2024-04-25T19:11:33.975724+0000","last_active":"2024-04-25T19:11:36.990390+0000","last_peered":"2024-04-25T19:11:36.990390+0000","last_clean":"2024-04-25T19:11:36.990390+0000","last_became_active":"2024-04-25T19:11:33.974921+0000","last_became_peered":"2024-04-25T19:11:33.974921+0000","last_unstale":"2024-04-25T19:11:36.990390+0000","last_undegraded":"2024-04-25T19:11:36.990390+0000","last_fullsized":"2024-04-25T19:11:36.990390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:55.208890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170918+0000","last_change":"2024-04-25T19:11:28.940100+0000","last_active":"2024-04-25T19:11:37.170918+0000","last_peered":"2024-04-25T19:11:37.170918+0000","last_clean":"2024-04-25T19:11:37.170918+0000","last_became_active":"2024-04-25T19:11:28.939482+0000","last_became_peered":"2024-04-25T19:11:28.939482+0000","last_unstale":"2024-04-25T19:11:37.170918+0000","last_undegraded":"2024-04-25T19:11:37.170918+0000","last_fullsized":"2024-04-25T19:11:37.170918+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962428+0000","last_change":"2024-04-25T19:11:29.945900+0000","last_active":"2024-04-25T19:11:34.962428+0000","last_peered":"2024-04-25T19:11:34.962428+0000","last_clean":"2024-04-25T19:11:34.962428+0000","last_became_active":"2024-04-25T19:11:29.945554+0000","last_became_peered":"2024-04-25T19:11:29.945554+0000","last_unstale":"2024-04-25T19:11:34.962428+0000","last_undegraded":"2024-04-25T19:11:34.962428+0000","last_fullsized":"2024-04-25T19:11:34.962428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990011+0000","last_change":"2024-04-25T19:11:33.975263+0000","last_active":"2024-04-25T19:11:36.990011+0000","last_peered":"2024-04-25T19:11:36.990011+0000","last_clean":"2024-04-25T19:11:36.990011+0000","last_became_active":"2024-04-25T19:11:33.974661+0000","last_became_peered":"2024-04-25T19:11:33.974661+0000","last_unstale":"2024-04-25T19:11:36.990011+0000","last_undegraded":"2024-04-25T19:11:36.990011+0000","last_fullsized":"2024-04-25T19:11:36.990011+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:29.019166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961851+0000","last_change":"2024-04-25T19:11:29.939828+0000","last_active":"2024-04-25T19:11:34.961851+0000","last_peered":"2024-04-25T19:11:34.961851+0000","last_clean":"2024-04-25T19:11:34.961851+0000","last_became_active":"2024-04-25T19:11:29.939513+0000","last_became_peered":"2024-04-25T19:11:29.939513+0000","last_unstale":"2024-04-25T19:11:34.961851+0000","last_undegraded":"2024-04-25T19:11:34.961851+0000","last_fullsized":"2024-04-25T19:11:34.961851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171950+0000","last_change":"2024-04-25T19:11:28.937339+0000","last_active":"2024-04-25T19:11:37.171950+0000","last_peered":"2024-04-25T19:11:37.171950+0000","last_clean":"2024-04-25T19:11:37.171950+0000","last_became_active":"2024-04-25T19:11:28.936962+0000","last_became_peered":"2024-04-25T19:11:28.936962+0000","last_unstale":"2024-04-25T19:11:37.171950+0000","last_undegraded":"2024-04-25T19:11:37.171950+0000","last_fullsized":"2024-04-25T19:11:37.171950+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:30.900062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990657+0000","last_change":"2024-04-25T19:11:33.975365+0000","last_active":"2024-04-25T19:11:36.990657+0000","last_peered":"2024-04-25T19:11:36.990657+0000","last_clean":"2024-04-25T19:11:36.990657+0000","last_became_active":"2024-04-25T19:11:33.974761+0000","last_became_peered":"2024-04-25T19:11:33.974761+0000","last_unstale":"2024-04-25T19:11:36.990657+0000","last_undegraded":"2024-04-25T19:11:36.990657+0000","last_fullsized":"2024-04-25T19:11:36.990657+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:46.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171339+0000","last_change":"2024-04-25T19:11:29.935255+0000","last_active":"2024-04-25T19:11:37.171339+0000","last_peered":"2024-04-25T19:11:37.171339+0000","last_clean":"2024-04-25T19:11:37.171339+0000","last_became_active":"2024-04-25T19:11:29.934911+0000","last_became_peered":"2024-04-25T19:11:29.934911+0000","last_unstale":"2024-04-25T19:11:37.171339+0000","last_undegraded":"2024-04-25T19:11:37.171339+0000","last_fullsized":"2024-04-25T19:11:37.171339+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470651+0000","last_change":"2024-04-25T19:11:28.937507+0000","last_active":"2024-04-25T19:11:35.470651+0000","last_peered":"2024-04-25T19:11:35.470651+0000","last_clean":"2024-04-25T19:11:35.470651+0000","last_became_active":"2024-04-25T19:11:28.936655+0000","last_became_peered":"2024-04-25T19:11:28.936655+0000","last_unstale":"2024-04-25T19:11:35.470651+0000","last_undegraded":"2024-04-25T19:11:35.470651+0000","last_fullsized":"2024-04-25T19:11:35.470651+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990392+0000","last_change":"2024-04-25T19:11:33.974529+0000","last_active":"2024-04-25T19:11:36.990392+0000","last_peered":"2024-04-25T19:11:36.990392+0000","last_clean":"2024-04-25T19:11:36.990392+0000","last_became_active":"2024-04-25T19:11:33.973907+0000","last_became_peered":"2024-04-25T19:11:33.973907+0000","last_unstale":"2024-04-25T19:11:36.990392+0000","last_undegraded":"2024-04-25T19:11:36.990392+0000","last_fullsized":"2024-04-25T19:11:36.990392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:52.041722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171394+0000","last_change":"2024-04-25T19:11:29.930361+0000","last_active":"2024-04-25T19:11:37.171394+0000","last_peered":"2024-04-25T19:11:37.171394+0000","last_clean":"2024-04-25T19:11:37.171394+0000","last_became_active":"2024-04-25T19:11:29.930060+0000","last_became_peered":"2024-04-25T19:11:29.930060+0000","last_unstale":"2024-04-25T19:11:37.171394+0000","last_undegraded":"2024-04-25T19:11:37.171394+0000","last_fullsized":"2024-04-25T19:11:37.171394+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470271+0000","last_change":"2024-04-25T19:11:28.930410+0000","last_active":"2024-04-25T19:11:35.470271+0000","last_peered":"2024-04-25T19:11:35.470271+0000","last_clean":"2024-04-25T19:11:35.470271+0000","last_became_active":"2024-04-25T19:11:28.930012+0000","last_became_peered":"2024-04-25T19:11:28.930012+0000","last_unstale":"2024-04-25T19:11:35.470271+0000","last_undegraded":"2024-04-25T19:11:35.470271+0000","last_fullsized":"2024-04-25T19:11:35.470271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.471102+0000","last_change":"2024-04-25T19:11:34.115931+0000","last_active":"2024-04-25T19:11:35.471102+0000","last_peered":"2024-04-25T19:11:35.471102+0000","last_clean":"2024-04-25T19:11:35.471102+0000","last_became_active":"2024-04-25T19:11:34.115519+0000","last_became_peered":"2024-04-25T19:11:34.115519+0000","last_unstale":"2024-04-25T19:11:35.471102+0000","last_undegraded":"2024-04-25T19:11:35.471102+0000","last_fullsized":"2024-04-25T19:11:35.471102+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.795134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990163+0000","last_change":"2024-04-25T19:11:29.930630+0000","last_active":"2024-04-25T19:11:36.990163+0000","last_peered":"2024-04-25T19:11:36.990163+0000","last_clean":"2024-04-25T19:11:36.990163+0000","last_became_active":"2024-04-25T19:11:29.930314+0000","last_became_peered":"2024-04-25T19:11:29.930314+0000","last_unstale":"2024-04-25T19:11:36.990163+0000","last_undegraded":"2024-04-25T19:11:36.990163+0000","last_fullsized":"2024-04-25T19:11:36.990163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:44.955421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170921+0000","last_change":"2024-04-25T19:11:28.938687+0000","last_active":"2024-04-25T19:11:37.170921+0000","last_peered":"2024-04-25T19:11:37.170921+0000","last_clean":"2024-04-25T19:11:37.170921+0000","last_became_active":"2024-04-25T19:11:28.938288+0000","last_became_peered":"2024-04-25T19:11:28.938288+0000","last_unstale":"2024-04-25T19:11:37.170921+0000","last_undegraded":"2024-04-25T19:11:37.170921+0000","last_fullsized":"2024-04-25T19:11:37.170921+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990666+0000","last_change":"2024-04-25T19:11:33.972056+0000","last_active":"2024-04-25T19:11:36.990666+0000","last_peered":"2024-04-25T19:11:36.990666+0000","last_clean":"2024-04-25T19:11:36.990666+0000","last_became_active":"2024-04-25T19:11:33.971484+0000","last_became_peered":"2024-04-25T19:11:33.971484+0000","last_unstale":"2024-04-25T19:11:36.990666+0000","last_undegraded":"2024-04-25T19:11:36.990666+0000","last_fullsized":"2024-04-25T19:11:36.990666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:52.951275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878644+0000","last_change":"2024-04-25T19:11:29.935281+0000","last_active":"2024-04-25T19:11:35.878644+0000","last_peered":"2024-04-25T19:11:35.878644+0000","last_clean":"2024-04-25T19:11:35.878644+0000","last_became_active":"2024-04-25T19:11:29.934928+0000","last_became_peered":"2024-04-25T19:11:29.934928+0000","last_unstale":"2024-04-25T19:11:35.878644+0000","last_undegraded":"2024-04-25T19:11:35.878644+0000","last_fullsized":"2024-04-25T19:11:35.878644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963243+0000","last_change":"2024-04-25T19:11:28.940436+0000","last_active":"2024-04-25T19:11:34.963243+0000","last_peered":"2024-04-25T19:11:34.963243+0000","last_clean":"2024-04-25T19:11:34.963243+0000","last_became_active":"2024-04-25T19:11:28.940050+0000","last_became_peered":"2024-04-25T19:11:28.940050+0000","last_unstale":"2024-04-25T19:11:34.963243+0000","last_undegraded":"2024-04-25T19:11:34.963243+0000","last_fullsized":"2024-04-25T19:11:34.963243+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989678+0000","last_change":"2024-04-25T19:11:34.115813+0000","last_active":"2024-04-25T19:11:36.989678+0000","last_peered":"2024-04-25T19:11:36.989678+0000","last_clean":"2024-04-25T19:11:36.989678+0000","last_became_active":"2024-04-25T19:11:34.115268+0000","last_became_peered":"2024-04-25T19:11:34.115268+0000","last_unstale":"2024-04-25T19:11:36.989678+0000","last_undegraded":"2024-04-25T19:11:36.989678+0000","last_fullsized":"2024-04-25T19:11:36.989678+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:54.194127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171971+0000","last_change":"2024-04-25T19:11:29.945249+0000","last_active":"2024-04-25T19:11:37.171971+0000","last_peered":"2024-04-25T19:11:37.171971+0000","last_clean":"2024-04-25T19:11:37.171971+0000","last_became_active":"2024-04-25T19:11:29.944930+0000","last_became_peered":"2024-04-25T19:11:29.944930+0000","last_unstale":"2024-04-25T19:11:37.171971+0000","last_undegraded":"2024-04-25T19:11:37.171971+0000","last_fullsized":"2024-04-25T19:11:37.171971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470980+0000","last_change":"2024-04-25T19:11:28.937963+0000","last_active":"2024-04-25T19:11:35.470980+0000","last_peered":"2024-04-25T19:11:35.470980+0000","last_clean":"2024-04-25T19:11:35.470980+0000","last_became_active":"2024-04-25T19:11:28.936818+0000","last_became_peered":"2024-04-25T19:11:28.936818+0000","last_unstale":"2024-04-25T19:11:35.470980+0000","last_undegraded":"2024-04-25T19:11:35.470980+0000","last_fullsized":"2024-04-25T19:11:35.470980+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170969+0000","last_change":"2024-04-25T19:11:34.113848+0000","last_active":"2024-04-25T19:11:37.170969+0000","last_peered":"2024-04-25T19:11:37.170969+0000","last_clean":"2024-04-25T19:11:37.170969+0000","last_became_active":"2024-04-25T19:11:34.113403+0000","last_became_peered":"2024-04-25T19:11:34.113403+0000","last_unstale":"2024-04-25T19:11:37.170969+0000","last_undegraded":"2024-04-25T19:11:37.170969+0000","last_fullsized":"2024-04-25T19:11:37.170969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:43.702771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470714+0000","last_change":"2024-04-25T19:11:29.930348+0000","last_active":"2024-04-25T19:11:35.470714+0000","last_peered":"2024-04-25T19:11:35.470714+0000","last_clean":"2024-04-25T19:11:35.470714+0000","last_became_active":"2024-04-25T19:11:29.930028+0000","last_became_peered":"2024-04-25T19:11:29.930028+0000","last_unstale":"2024-04-25T19:11:35.470714+0000","last_undegraded":"2024-04-25T19:11:35.470714+0000","last_fullsized":"2024-04-25T19:11:35.470714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170925+0000","last_change":"2024-04-25T19:11:28.939508+0000","last_active":"2024-04-25T19:11:37.170925+0000","last_peered":"2024-04-25T19:11:37.170925+0000","last_clean":"2024-04-25T19:11:37.170925+0000","last_became_active":"2024-04-25T19:11:28.939050+0000","last_became_peered":"2024-04-25T19:11:28.939050+0000","last_unstale":"2024-04-25T19:11:37.170925+0000","last_undegraded":"2024-04-25T19:11:37.170925+0000","last_fullsized":"2024-04-25T19:11:37.170925+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:58:00.743906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878165+0000","last_change":"2024-04-25T19:11:34.110317+0000","last_active":"2024-04-25T19:11:35.878165+0000","last_peered":"2024-04-25T19:11:35.878165+0000","last_clean":"2024-04-25T19:11:35.878165+0000","last_became_active":"2024-04-25T19:11:34.109906+0000","last_became_peered":"2024-04-25T19:11:34.109906+0000","last_unstale":"2024-04-25T19:11:35.878165+0000","last_undegraded":"2024-04-25T19:11:35.878165+0000","last_fullsized":"2024-04-25T19:11:35.878165+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:35.687485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962541+0000","last_change":"2024-04-25T19:11:29.952079+0000","last_active":"2024-04-25T19:11:34.962541+0000","last_peered":"2024-04-25T19:11:34.962541+0000","last_clean":"2024-04-25T19:11:34.962541+0000","last_became_active":"2024-04-25T19:11:29.951437+0000","last_became_peered":"2024-04-25T19:11:29.951437+0000","last_unstale":"2024-04-25T19:11:34.962541+0000","last_undegraded":"2024-04-25T19:11:34.962541+0000","last_fullsized":"2024-04-25T19:11:34.962541+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470828+0000","last_change":"2024-04-25T19:11:28.933142+0000","last_active":"2024-04-25T19:11:35.470828+0000","last_peered":"2024-04-25T19:11:35.470828+0000","last_clean":"2024-04-25T19:11:35.470828+0000","last_became_active":"2024-04-25T19:11:28.932682+0000","last_became_peered":"2024-04-25T19:11:28.932682+0000","last_unstale":"2024-04-25T19:11:35.470828+0000","last_undegraded":"2024-04-25T19:11:35.470828+0000","last_fullsized":"2024-04-25T19:11:35.470828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:38.236609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470450+0000","last_change":"2024-04-25T19:11:33.971859+0000","last_active":"2024-04-25T19:11:35.470450+0000","last_peered":"2024-04-25T19:11:35.470450+0000","last_clean":"2024-04-25T19:11:35.470450+0000","last_became_active":"2024-04-25T19:11:33.971349+0000","last_became_peered":"2024-04-25T19:11:33.971349+0000","last_unstale":"2024-04-25T19:11:35.470450+0000","last_undegraded":"2024-04-25T19:11:35.470450+0000","last_fullsized":"2024-04-25T19:11:35.470450+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:43.916700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990771+0000","last_change":"2024-04-25T19:11:29.929623+0000","last_active":"2024-04-25T19:11:36.990771+0000","last_peered":"2024-04-25T19:11:36.990771+0000","last_clean":"2024-04-25T19:11:36.990771+0000","last_became_active":"2024-04-25T19:11:29.929306+0000","last_became_peered":"2024-04-25T19:11:29.929306+0000","last_unstale":"2024-04-25T19:11:36.990771+0000","last_undegraded":"2024-04-25T19:11:36.990771+0000","last_fullsized":"2024-04-25T19:11:36.990771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:49.958865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963468+0000","last_change":"2024-04-25T19:11:28.933056+0000","last_active":"2024-04-25T19:11:34.963468+0000","last_peered":"2024-04-25T19:11:34.963468+0000","last_clean":"2024-04-25T19:11:34.963468+0000","last_became_active":"2024-04-25T19:11:28.930233+0000","last_became_peered":"2024-04-25T19:11:28.930233+0000","last_unstale":"2024-04-25T19:11:34.963468+0000","last_undegraded":"2024-04-25T19:11:34.963468+0000","last_fullsized":"2024-04-25T19:11:34.963468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470757+0000","last_change":"2024-04-25T19:11:33.972447+0000","last_active":"2024-04-25T19:11:35.470757+0000","last_peered":"2024-04-25T19:11:35.470757+0000","last_clean":"2024-04-25T19:11:35.470757+0000","last_became_active":"2024-04-25T19:11:33.970645+0000","last_became_peered":"2024-04-25T19:11:33.970645+0000","last_unstale":"2024-04-25T19:11:35.470757+0000","last_undegraded":"2024-04-25T19:11:35.470757+0000","last_fullsized":"2024-04-25T19:11:35.470757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:40.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470256+0000","last_change":"2024-04-25T19:11:29.930967+0000","last_active":"2024-04-25T19:11:35.470256+0000","last_peered":"2024-04-25T19:11:35.470256+0000","last_clean":"2024-04-25T19:11:35.470256+0000","last_became_active":"2024-04-25T19:11:29.930682+0000","last_became_peered":"2024-04-25T19:11:29.930682+0000","last_unstale":"2024-04-25T19:11:35.470256+0000","last_undegraded":"2024-04-25T19:11:35.470256+0000","last_fullsized":"2024-04-25T19:11:35.470256+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960304+0000","last_change":"2024-04-25T19:11:28.932205+0000","last_active":"2024-04-25T19:11:34.960304+0000","last_peered":"2024-04-25T19:11:34.960304+0000","last_clean":"2024-04-25T19:11:34.960304+0000","last_became_active":"2024-04-25T19:11:28.931851+0000","last_became_peered":"2024-04-25T19:11:28.931851+0000","last_unstale":"2024-04-25T19:11:34.960304+0000","last_undegraded":"2024-04-25T19:11:34.960304+0000","last_fullsized":"2024-04-25T19:11:34.960304+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.471128+0000","last_change":"2024-04-25T19:11:33.971485+0000","last_active":"2024-04-25T19:11:35.471128+0000","last_peered":"2024-04-25T19:11:35.471128+0000","last_clean":"2024-04-25T19:11:35.471128+0000","last_became_active":"2024-04-25T19:11:33.971022+0000","last_became_peered":"2024-04-25T19:11:33.971022+0000","last_unstale":"2024-04-25T19:11:35.471128+0000","last_undegraded":"2024-04-25T19:11:35.471128+0000","last_fullsized":"2024-04-25T19:11:35.471128+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:38.690554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878004+0000","last_change":"2024-04-25T19:11:29.936287+0000","last_active":"2024-04-25T19:11:35.878004+0000","last_peered":"2024-04-25T19:11:35.878004+0000","last_clean":"2024-04-25T19:11:35.878004+0000","last_became_active":"2024-04-25T19:11:29.935945+0000","last_became_peered":"2024-04-25T19:11:29.935945+0000","last_unstale":"2024-04-25T19:11:35.878004+0000","last_undegraded":"2024-04-25T19:11:35.878004+0000","last_fullsized":"2024-04-25T19:11:35.878004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:41.846903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170920+0000","last_change":"2024-04-25T19:11:28.940966+0000","last_active":"2024-04-25T19:11:37.170920+0000","last_peered":"2024-04-25T19:11:37.170920+0000","last_clean":"2024-04-25T19:11:37.170920+0000","last_became_active":"2024-04-25T19:11:28.939773+0000","last_became_peered":"2024-04-25T19:11:28.939773+0000","last_unstale":"2024-04-25T19:11:37.170920+0000","last_undegraded":"2024-04-25T19:11:37.170920+0000","last_fullsized":"2024-04-25T19:11:37.170920+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878537+0000","last_change":"2024-04-25T19:11:34.113282+0000","last_active":"2024-04-25T19:11:35.878537+0000","last_peered":"2024-04-25T19:11:35.878537+0000","last_clean":"2024-04-25T19:11:35.878537+0000","last_became_active":"2024-04-25T19:11:34.112444+0000","last_became_peered":"2024-04-25T19:11:34.112444+0000","last_unstale":"2024-04-25T19:11:35.878537+0000","last_undegraded":"2024-04-25T19:11:35.878537+0000","last_fullsized":"2024-04-25T19:11:35.878537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:07:43.702771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170832+0000","last_change":"2024-04-25T19:11:29.949528+0000","last_active":"2024-04-25T19:11:37.170832+0000","last_peered":"2024-04-25T19:11:37.170832+0000","last_clean":"2024-04-25T19:11:37.170832+0000","last_became_active":"2024-04-25T19:11:29.948760+0000","last_became_peered":"2024-04-25T19:11:29.948760+0000","last_unstale":"2024-04-25T19:11:37.170832+0000","last_undegraded":"2024-04-25T19:11:37.170832+0000","last_fullsized":"2024-04-25T19:11:37.170832+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963446+0000","last_change":"2024-04-25T19:11:28.928976+0000","last_active":"2024-04-25T19:11:34.963446+0000","last_peered":"2024-04-25T19:11:34.963446+0000","last_clean":"2024-04-25T19:11:34.963446+0000","last_became_active":"2024-04-25T19:11:28.928563+0000","last_became_peered":"2024-04-25T19:11:28.928563+0000","last_unstale":"2024-04-25T19:11:34.963446+0000","last_undegraded":"2024-04-25T19:11:34.963446+0000","last_fullsized":"2024-04-25T19:11:34.963446+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":117,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171444+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:11:37.171444+0000","last_peered":"2024-04-25T19:11:37.171444+0000","last_clean":"2024-04-25T19:11:37.171444+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:11:37.171444+0000","last_undegraded":"2024-04-25T19:11:37.171444+0000","last_fullsized":"2024-04-25T19:11:37.171444+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962407+0000","last_change":"2024-04-25T19:11:34.113394+0000","last_active":"2024-04-25T19:11:34.962407+0000","last_peered":"2024-04-25T19:11:34.962407+0000","last_clean":"2024-04-25T19:11:34.962407+0000","last_became_active":"2024-04-25T19:11:34.112950+0000","last_became_peered":"2024-04-25T19:11:34.112950+0000","last_unstale":"2024-04-25T19:11:34.962407+0000","last_undegraded":"2024-04-25T19:11:34.962407+0000","last_fullsized":"2024-04-25T19:11:34.962407+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:43.916700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171981+0000","last_change":"2024-04-25T19:11:29.946818+0000","last_active":"2024-04-25T19:11:37.171981+0000","last_peered":"2024-04-25T19:11:37.171981+0000","last_clean":"2024-04-25T19:11:37.171981+0000","last_became_active":"2024-04-25T19:11:29.946517+0000","last_became_peered":"2024-04-25T19:11:29.946517+0000","last_unstale":"2024-04-25T19:11:37.171981+0000","last_undegraded":"2024-04-25T19:11:37.171981+0000","last_fullsized":"2024-04-25T19:11:37.171981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960736+0000","last_change":"2024-04-25T19:11:28.928753+0000","last_active":"2024-04-25T19:11:34.960736+0000","last_peered":"2024-04-25T19:11:34.960736+0000","last_clean":"2024-04-25T19:11:34.960736+0000","last_became_active":"2024-04-25T19:11:28.928388+0000","last_became_peered":"2024-04-25T19:11:28.928388+0000","last_unstale":"2024-04-25T19:11:34.960736+0000","last_undegraded":"2024-04-25T19:11:34.960736+0000","last_fullsized":"2024-04-25T19:11:34.960736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961773+0000","last_change":"2024-04-25T19:11:34.110545+0000","last_active":"2024-04-25T19:11:34.961773+0000","last_peered":"2024-04-25T19:11:34.961773+0000","last_clean":"2024-04-25T19:11:34.961773+0000","last_became_active":"2024-04-25T19:11:34.110024+0000","last_became_peered":"2024-04-25T19:11:34.110024+0000","last_unstale":"2024-04-25T19:11:34.961773+0000","last_undegraded":"2024-04-25T19:11:34.961773+0000","last_fullsized":"2024-04-25T19:11:34.961773+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:46.755893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962527+0000","last_change":"2024-04-25T19:11:29.950030+0000","last_active":"2024-04-25T19:11:34.962527+0000","last_peered":"2024-04-25T19:11:34.962527+0000","last_clean":"2024-04-25T19:11:34.962527+0000","last_became_active":"2024-04-25T19:11:29.949424+0000","last_became_peered":"2024-04-25T19:11:29.949424+0000","last_unstale":"2024-04-25T19:11:34.962527+0000","last_undegraded":"2024-04-25T19:11:34.962527+0000","last_fullsized":"2024-04-25T19:11:34.962527+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:48.917939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470828+0000","last_change":"2024-04-25T19:11:28.934208+0000","last_active":"2024-04-25T19:11:35.470828+0000","last_peered":"2024-04-25T19:11:35.470828+0000","last_clean":"2024-04-25T19:11:35.470828+0000","last_became_active":"2024-04-25T19:11:28.933866+0000","last_became_peered":"2024-04-25T19:11:28.933866+0000","last_unstale":"2024-04-25T19:11:35.470828+0000","last_undegraded":"2024-04-25T19:11:35.470828+0000","last_fullsized":"2024-04-25T19:11:35.470828+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:30.900062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878372+0000","last_change":"2024-04-25T19:11:33.980318+0000","last_active":"2024-04-25T19:11:35.878372+0000","last_peered":"2024-04-25T19:11:35.878372+0000","last_clean":"2024-04-25T19:11:35.878372+0000","last_became_active":"2024-04-25T19:11:33.979818+0000","last_became_peered":"2024-04-25T19:11:33.979818+0000","last_unstale":"2024-04-25T19:11:35.878372+0000","last_undegraded":"2024-04-25T19:11:35.878372+0000","last_fullsized":"2024-04-25T19:11:35.878372+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:18:38.690554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172468+0000","last_change":"2024-04-25T19:11:29.940479+0000","last_active":"2024-04-25T19:11:37.172468+0000","last_peered":"2024-04-25T19:11:37.172468+0000","last_clean":"2024-04-25T19:11:37.172468+0000","last_became_active":"2024-04-25T19:11:29.940140+0000","last_became_peered":"2024-04-25T19:11:29.940140+0000","last_unstale":"2024-04-25T19:11:37.172468+0000","last_undegraded":"2024-04-25T19:11:37.172468+0000","last_fullsized":"2024-04-25T19:11:37.172468+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:47:56.761797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960699+0000","last_change":"2024-04-25T19:11:28.933352+0000","last_active":"2024-04-25T19:11:34.960699+0000","last_peered":"2024-04-25T19:11:34.960699+0000","last_clean":"2024-04-25T19:11:34.960699+0000","last_became_active":"2024-04-25T19:11:28.933016+0000","last_became_peered":"2024-04-25T19:11:28.933016+0000","last_unstale":"2024-04-25T19:11:34.960699+0000","last_undegraded":"2024-04-25T19:11:34.960699+0000","last_fullsized":"2024-04-25T19:11:34.960699+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961463+0000","last_change":"2024-04-25T19:11:34.110567+0000","last_active":"2024-04-25T19:11:34.961463+0000","last_peered":"2024-04-25T19:11:34.961463+0000","last_clean":"2024-04-25T19:11:34.961463+0000","last_became_active":"2024-04-25T19:11:34.110074+0000","last_became_peered":"2024-04-25T19:11:34.110074+0000","last_unstale":"2024-04-25T19:11:34.961463+0000","last_undegraded":"2024-04-25T19:11:34.961463+0000","last_fullsized":"2024-04-25T19:11:34.961463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:54.738252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172199+0000","last_change":"2024-04-25T19:11:29.939683+0000","last_active":"2024-04-25T19:11:37.172199+0000","last_peered":"2024-04-25T19:11:37.172199+0000","last_clean":"2024-04-25T19:11:37.172199+0000","last_became_active":"2024-04-25T19:11:29.939372+0000","last_became_peered":"2024-04-25T19:11:29.939372+0000","last_unstale":"2024-04-25T19:11:37.172199+0000","last_undegraded":"2024-04-25T19:11:37.172199+0000","last_fullsized":"2024-04-25T19:11:37.172199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:08:51.175553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989410+0000","last_change":"2024-04-25T19:11:28.928627+0000","last_active":"2024-04-25T19:11:36.989410+0000","last_peered":"2024-04-25T19:11:36.989410+0000","last_clean":"2024-04-25T19:11:36.989410+0000","last_became_active":"2024-04-25T19:11:28.928016+0000","last_became_peered":"2024-04-25T19:11:28.928016+0000","last_unstale":"2024-04-25T19:11:36.989410+0000","last_undegraded":"2024-04-25T19:11:36.989410+0000","last_fullsized":"2024-04-25T19:11:36.989410+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:30:16.833337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.989757+0000","last_change":"2024-04-25T19:11:33.971762+0000","last_active":"2024-04-25T19:11:36.989757+0000","last_peered":"2024-04-25T19:11:36.989757+0000","last_clean":"2024-04-25T19:11:36.989757+0000","last_became_active":"2024-04-25T19:11:33.971317+0000","last_became_peered":"2024-04-25T19:11:33.971317+0000","last_unstale":"2024-04-25T19:11:36.989757+0000","last_undegraded":"2024-04-25T19:11:36.989757+0000","last_fullsized":"2024-04-25T19:11:36.989757+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:23.748216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171561+0000","last_change":"2024-04-25T19:11:29.938906+0000","last_active":"2024-04-25T19:11:37.171561+0000","last_peered":"2024-04-25T19:11:37.171561+0000","last_clean":"2024-04-25T19:11:37.171561+0000","last_became_active":"2024-04-25T19:11:29.938575+0000","last_became_peered":"2024-04-25T19:11:29.938575+0000","last_unstale":"2024-04-25T19:11:37.171561+0000","last_undegraded":"2024-04-25T19:11:37.171561+0000","last_fullsized":"2024-04-25T19:11:37.171561+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960132+0000","last_change":"2024-04-25T19:11:28.931773+0000","last_active":"2024-04-25T19:11:34.960132+0000","last_peered":"2024-04-25T19:11:34.960132+0000","last_clean":"2024-04-25T19:11:34.960132+0000","last_became_active":"2024-04-25T19:11:28.931409+0000","last_became_peered":"2024-04-25T19:11:28.931409+0000","last_unstale":"2024-04-25T19:11:34.960132+0000","last_undegraded":"2024-04-25T19:11:34.960132+0000","last_fullsized":"2024-04-25T19:11:34.960132+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:27:30.904833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878361+0000","last_change":"2024-04-25T19:11:33.975482+0000","last_active":"2024-04-25T19:11:35.878361+0000","last_peered":"2024-04-25T19:11:35.878361+0000","last_clean":"2024-04-25T19:11:35.878361+0000","last_became_active":"2024-04-25T19:11:33.975042+0000","last_became_peered":"2024-04-25T19:11:33.975042+0000","last_unstale":"2024-04-25T19:11:35.878361+0000","last_undegraded":"2024-04-25T19:11:35.878361+0000","last_fullsized":"2024-04-25T19:11:35.878361+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:00.795134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961644+0000","last_change":"2024-04-25T19:11:29.943783+0000","last_active":"2024-04-25T19:11:34.961644+0000","last_peered":"2024-04-25T19:11:34.961644+0000","last_clean":"2024-04-25T19:11:34.961644+0000","last_became_active":"2024-04-25T19:11:29.942647+0000","last_became_peered":"2024-04-25T19:11:29.942647+0000","last_unstale":"2024-04-25T19:11:34.961644+0000","last_undegraded":"2024-04-25T19:11:34.961644+0000","last_fullsized":"2024-04-25T19:11:34.961644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961794+0000","last_change":"2024-04-25T19:11:28.934715+0000","last_active":"2024-04-25T19:11:34.961794+0000","last_peered":"2024-04-25T19:11:34.961794+0000","last_clean":"2024-04-25T19:11:34.961794+0000","last_became_active":"2024-04-25T19:11:28.934331+0000","last_became_peered":"2024-04-25T19:11:28.934331+0000","last_unstale":"2024-04-25T19:11:34.961794+0000","last_undegraded":"2024-04-25T19:11:34.961794+0000","last_fullsized":"2024-04-25T19:11:34.961794+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171897+0000","last_change":"2024-04-25T19:11:34.115885+0000","last_active":"2024-04-25T19:11:37.171897+0000","last_peered":"2024-04-25T19:11:37.171897+0000","last_clean":"2024-04-25T19:11:37.171897+0000","last_became_active":"2024-04-25T19:11:34.115489+0000","last_became_peered":"2024-04-25T19:11:34.115489+0000","last_unstale":"2024-04-25T19:11:37.171897+0000","last_undegraded":"2024-04-25T19:11:37.171897+0000","last_fullsized":"2024-04-25T19:11:37.171897+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:20:15.088779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171931+0000","last_change":"2024-04-25T19:11:29.937605+0000","last_active":"2024-04-25T19:11:37.171931+0000","last_peered":"2024-04-25T19:11:37.171931+0000","last_clean":"2024-04-25T19:11:37.171931+0000","last_became_active":"2024-04-25T19:11:29.937045+0000","last_became_peered":"2024-04-25T19:11:29.937045+0000","last_unstale":"2024-04-25T19:11:37.171931+0000","last_undegraded":"2024-04-25T19:11:37.171931+0000","last_fullsized":"2024-04-25T19:11:37.171931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:39.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963413+0000","last_change":"2024-04-25T19:11:28.937550+0000","last_active":"2024-04-25T19:11:34.963413+0000","last_peered":"2024-04-25T19:11:34.963413+0000","last_clean":"2024-04-25T19:11:34.963413+0000","last_became_active":"2024-04-25T19:11:28.937224+0000","last_became_peered":"2024-04-25T19:11:28.937224+0000","last_unstale":"2024-04-25T19:11:34.963413+0000","last_undegraded":"2024-04-25T19:11:34.963413+0000","last_fullsized":"2024-04-25T19:11:34.963413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171513+0000","last_change":"2024-04-25T19:11:34.112538+0000","last_active":"2024-04-25T19:11:37.171513+0000","last_peered":"2024-04-25T19:11:37.171513+0000","last_clean":"2024-04-25T19:11:37.171513+0000","last_became_active":"2024-04-25T19:11:34.112105+0000","last_became_peered":"2024-04-25T19:11:34.112105+0000","last_unstale":"2024-04-25T19:11:37.171513+0000","last_undegraded":"2024-04-25T19:11:37.171513+0000","last_fullsized":"2024-04-25T19:11:37.171513+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:04.753913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878413+0000","last_change":"2024-04-25T19:11:29.940791+0000","last_active":"2024-04-25T19:11:35.878413+0000","last_peered":"2024-04-25T19:11:35.878413+0000","last_clean":"2024-04-25T19:11:35.878413+0000","last_became_active":"2024-04-25T19:11:29.940494+0000","last_became_peered":"2024-04-25T19:11:29.940494+0000","last_unstale":"2024-04-25T19:11:35.878413+0000","last_undegraded":"2024-04-25T19:11:35.878413+0000","last_fullsized":"2024-04-25T19:11:35.878413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878662+0000","last_change":"2024-04-25T19:11:28.931517+0000","last_active":"2024-04-25T19:11:35.878662+0000","last_peered":"2024-04-25T19:11:35.878662+0000","last_clean":"2024-04-25T19:11:35.878662+0000","last_became_active":"2024-04-25T19:11:28.930964+0000","last_became_peered":"2024-04-25T19:11:28.930964+0000","last_unstale":"2024-04-25T19:11:35.878662+0000","last_undegraded":"2024-04-25T19:11:35.878662+0000","last_fullsized":"2024-04-25T19:11:35.878662+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:10:14.977696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962359+0000","last_change":"2024-04-25T19:11:34.113386+0000","last_active":"2024-04-25T19:11:34.962359+0000","last_peered":"2024-04-25T19:11:34.962359+0000","last_clean":"2024-04-25T19:11:34.962359+0000","last_became_active":"2024-04-25T19:11:34.112820+0000","last_became_peered":"2024-04-25T19:11:34.112820+0000","last_unstale":"2024-04-25T19:11:34.962359+0000","last_undegraded":"2024-04-25T19:11:34.962359+0000","last_fullsized":"2024-04-25T19:11:34.962359+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:48:04.753913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171892+0000","last_change":"2024-04-25T19:11:29.933966+0000","last_active":"2024-04-25T19:11:37.171892+0000","last_peered":"2024-04-25T19:11:37.171892+0000","last_clean":"2024-04-25T19:11:37.171892+0000","last_became_active":"2024-04-25T19:11:29.933549+0000","last_became_peered":"2024-04-25T19:11:29.933549+0000","last_unstale":"2024-04-25T19:11:37.171892+0000","last_undegraded":"2024-04-25T19:11:37.171892+0000","last_fullsized":"2024-04-25T19:11:37.171892+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:39.517432+0000","last_change":"2024-04-25T19:11:28.936547+0000","last_active":"2024-04-25T19:11:39.517432+0000","last_peered":"2024-04-25T19:11:39.517432+0000","last_clean":"2024-04-25T19:11:39.517432+0000","last_became_active":"2024-04-25T19:11:28.935816+0000","last_became_peered":"2024-04-25T19:11:28.935816+0000","last_unstale":"2024-04-25T19:11:39.517432+0000","last_undegraded":"2024-04-25T19:11:39.517432+0000","last_fullsized":"2024-04-25T19:11:39.517432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470655+0000","last_change":"2024-04-25T19:11:33.972776+0000","last_active":"2024-04-25T19:11:35.470655+0000","last_peered":"2024-04-25T19:11:35.470655+0000","last_clean":"2024-04-25T19:11:35.470655+0000","last_became_active":"2024-04-25T19:11:33.971757+0000","last_became_peered":"2024-04-25T19:11:33.971757+0000","last_unstale":"2024-04-25T19:11:35.470655+0000","last_undegraded":"2024-04-25T19:11:35.470655+0000","last_fullsized":"2024-04-25T19:11:35.470655+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:35.687485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990580+0000","last_change":"2024-04-25T19:11:29.925691+0000","last_active":"2024-04-25T19:11:36.990580+0000","last_peered":"2024-04-25T19:11:36.990580+0000","last_clean":"2024-04-25T19:11:36.990580+0000","last_became_active":"2024-04-25T19:11:29.924990+0000","last_became_peered":"2024-04-25T19:11:29.924990+0000","last_unstale":"2024-04-25T19:11:36.990580+0000","last_undegraded":"2024-04-25T19:11:36.990580+0000","last_fullsized":"2024-04-25T19:11:36.990580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:39.242588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470697+0000","last_change":"2024-04-25T19:11:28.937077+0000","last_active":"2024-04-25T19:11:35.470697+0000","last_peered":"2024-04-25T19:11:35.470697+0000","last_clean":"2024-04-25T19:11:35.470697+0000","last_became_active":"2024-04-25T19:11:28.936181+0000","last_became_peered":"2024-04-25T19:11:28.936181+0000","last_unstale":"2024-04-25T19:11:35.470697+0000","last_undegraded":"2024-04-25T19:11:35.470697+0000","last_fullsized":"2024-04-25T19:11:35.470697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990233+0000","last_change":"2024-04-25T19:11:33.972375+0000","last_active":"2024-04-25T19:11:36.990233+0000","last_peered":"2024-04-25T19:11:36.990233+0000","last_clean":"2024-04-25T19:11:36.990233+0000","last_became_active":"2024-04-25T19:11:33.971792+0000","last_became_peered":"2024-04-25T19:11:33.971792+0000","last_unstale":"2024-04-25T19:11:36.990233+0000","last_undegraded":"2024-04-25T19:11:36.990233+0000","last_fullsized":"2024-04-25T19:11:36.990233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:13.742925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878422+0000","last_change":"2024-04-25T19:11:29.943424+0000","last_active":"2024-04-25T19:11:35.878422+0000","last_peered":"2024-04-25T19:11:35.878422+0000","last_clean":"2024-04-25T19:11:35.878422+0000","last_became_active":"2024-04-25T19:11:29.943096+0000","last_became_peered":"2024-04-25T19:11:29.943096+0000","last_unstale":"2024-04-25T19:11:35.878422+0000","last_undegraded":"2024-04-25T19:11:35.878422+0000","last_fullsized":"2024-04-25T19:11:35.878422+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962894+0000","last_change":"2024-04-25T19:11:28.934405+0000","last_active":"2024-04-25T19:11:34.962894+0000","last_peered":"2024-04-25T19:11:34.962894+0000","last_clean":"2024-04-25T19:11:34.962894+0000","last_became_active":"2024-04-25T19:11:28.933743+0000","last_became_peered":"2024-04-25T19:11:28.933743+0000","last_unstale":"2024-04-25T19:11:34.962894+0000","last_undegraded":"2024-04-25T19:11:34.962894+0000","last_fullsized":"2024-04-25T19:11:34.962894+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:28:13.354535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171201+0000","last_change":"2024-04-25T19:11:33.969689+0000","last_active":"2024-04-25T19:11:37.171201+0000","last_peered":"2024-04-25T19:11:37.171201+0000","last_clean":"2024-04-25T19:11:37.171201+0000","last_became_active":"2024-04-25T19:11:33.969257+0000","last_became_peered":"2024-04-25T19:11:33.969257+0000","last_unstale":"2024-04-25T19:11:37.171201+0000","last_undegraded":"2024-04-25T19:11:37.171201+0000","last_fullsized":"2024-04-25T19:11:37.171201+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:57.736550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962757+0000","last_change":"2024-04-25T19:11:29.949944+0000","last_active":"2024-04-25T19:11:34.962757+0000","last_peered":"2024-04-25T19:11:34.962757+0000","last_clean":"2024-04-25T19:11:34.962757+0000","last_became_active":"2024-04-25T19:11:29.949521+0000","last_became_peered":"2024-04-25T19:11:29.949521+0000","last_unstale":"2024-04-25T19:11:34.962757+0000","last_undegraded":"2024-04-25T19:11:34.962757+0000","last_fullsized":"2024-04-25T19:11:34.962757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:25:48.008386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963008+0000","last_change":"2024-04-25T19:11:28.942291+0000","last_active":"2024-04-25T19:11:34.963008+0000","last_peered":"2024-04-25T19:11:34.963008+0000","last_clean":"2024-04-25T19:11:34.963008+0000","last_became_active":"2024-04-25T19:11:28.941951+0000","last_became_peered":"2024-04-25T19:11:28.941951+0000","last_unstale":"2024-04-25T19:11:34.963008+0000","last_undegraded":"2024-04-25T19:11:34.963008+0000","last_fullsized":"2024-04-25T19:11:34.963008+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.952886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171184+0000","last_change":"2024-04-25T19:11:34.114829+0000","last_active":"2024-04-25T19:11:37.171184+0000","last_peered":"2024-04-25T19:11:37.171184+0000","last_clean":"2024-04-25T19:11:37.171184+0000","last_became_active":"2024-04-25T19:11:34.114452+0000","last_became_peered":"2024-04-25T19:11:34.114452+0000","last_unstale":"2024-04-25T19:11:37.171184+0000","last_undegraded":"2024-04-25T19:11:37.171184+0000","last_fullsized":"2024-04-25T19:11:37.171184+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:35:54.738252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878339+0000","last_change":"2024-04-25T19:11:34.114794+0000","last_active":"2024-04-25T19:11:35.878339+0000","last_peered":"2024-04-25T19:11:35.878339+0000","last_clean":"2024-04-25T19:11:35.878339+0000","last_became_active":"2024-04-25T19:11:34.114376+0000","last_became_peered":"2024-04-25T19:11:34.114376+0000","last_unstale":"2024-04-25T19:11:35.878339+0000","last_undegraded":"2024-04-25T19:11:35.878339+0000","last_fullsized":"2024-04-25T19:11:35.878339+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:40.027207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470153+0000","last_change":"2024-04-25T19:11:29.935850+0000","last_active":"2024-04-25T19:11:35.470153+0000","last_peered":"2024-04-25T19:11:35.470153+0000","last_clean":"2024-04-25T19:11:35.470153+0000","last_became_active":"2024-04-25T19:11:29.935546+0000","last_became_peered":"2024-04-25T19:11:29.935546+0000","last_unstale":"2024-04-25T19:11:35.470153+0000","last_undegraded":"2024-04-25T19:11:35.470153+0000","last_fullsized":"2024-04-25T19:11:35.470153+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961554+0000","last_change":"2024-04-25T19:11:28.928204+0000","last_active":"2024-04-25T19:11:34.961554+0000","last_peered":"2024-04-25T19:11:34.961554+0000","last_clean":"2024-04-25T19:11:34.961554+0000","last_became_active":"2024-04-25T19:11:28.927812+0000","last_became_peered":"2024-04-25T19:11:28.927812+0000","last_unstale":"2024-04-25T19:11:34.961554+0000","last_undegraded":"2024-04-25T19:11:34.961554+0000","last_fullsized":"2024-04-25T19:11:34.961554+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:14.021575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961976+0000","last_change":"2024-04-25T19:11:33.971937+0000","last_active":"2024-04-25T19:11:34.961976+0000","last_peered":"2024-04-25T19:11:34.961976+0000","last_clean":"2024-04-25T19:11:34.961976+0000","last_became_active":"2024-04-25T19:11:33.971297+0000","last_became_peered":"2024-04-25T19:11:33.971297+0000","last_unstale":"2024-04-25T19:11:34.961976+0000","last_undegraded":"2024-04-25T19:11:34.961976+0000","last_fullsized":"2024-04-25T19:11:34.961976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:52.951275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171272+0000","last_change":"2024-04-25T19:11:29.936690+0000","last_active":"2024-04-25T19:11:37.171272+0000","last_peered":"2024-04-25T19:11:37.171272+0000","last_clean":"2024-04-25T19:11:37.171272+0000","last_became_active":"2024-04-25T19:11:29.934968+0000","last_became_peered":"2024-04-25T19:11:29.934968+0000","last_unstale":"2024-04-25T19:11:37.171272+0000","last_undegraded":"2024-04-25T19:11:37.171272+0000","last_fullsized":"2024-04-25T19:11:37.171272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:30:24.985830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170564+0000","last_change":"2024-04-25T19:11:28.936664+0000","last_active":"2024-04-25T19:11:37.170564+0000","last_peered":"2024-04-25T19:11:37.170564+0000","last_clean":"2024-04-25T19:11:37.170564+0000","last_became_active":"2024-04-25T19:11:28.936236+0000","last_became_peered":"2024-04-25T19:11:28.936236+0000","last_unstale":"2024-04-25T19:11:37.170564+0000","last_undegraded":"2024-04-25T19:11:37.170564+0000","last_fullsized":"2024-04-25T19:11:37.170564+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:14:31.577289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962069+0000","last_change":"2024-04-25T19:11:28.934644+0000","last_active":"2024-04-25T19:11:34.962069+0000","last_peered":"2024-04-25T19:11:34.962069+0000","last_clean":"2024-04-25T19:11:34.962069+0000","last_became_active":"2024-04-25T19:11:28.934282+0000","last_became_peered":"2024-04-25T19:11:28.934282+0000","last_unstale":"2024-04-25T19:11:34.962069+0000","last_undegraded":"2024-04-25T19:11:34.962069+0000","last_fullsized":"2024-04-25T19:11:34.962069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:28:43.949443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878411+0000","last_change":"2024-04-25T19:11:29.934419+0000","last_active":"2024-04-25T19:11:35.878411+0000","last_peered":"2024-04-25T19:11:35.878411+0000","last_clean":"2024-04-25T19:11:35.878411+0000","last_became_active":"2024-04-25T19:11:29.934119+0000","last_became_peered":"2024-04-25T19:11:29.934119+0000","last_unstale":"2024-04-25T19:11:35.878411+0000","last_undegraded":"2024-04-25T19:11:35.878411+0000","last_fullsized":"2024-04-25T19:11:35.878411+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960537+0000","last_change":"2024-04-25T19:11:33.969976+0000","last_active":"2024-04-25T19:11:34.960537+0000","last_peered":"2024-04-25T19:11:34.960537+0000","last_clean":"2024-04-25T19:11:34.960537+0000","last_became_active":"2024-04-25T19:11:33.969493+0000","last_became_peered":"2024-04-25T19:11:33.969493+0000","last_unstale":"2024-04-25T19:11:34.960537+0000","last_undegraded":"2024-04-25T19:11:34.960537+0000","last_fullsized":"2024-04-25T19:11:34.960537+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:20:45.880240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171502+0000","last_change":"2024-04-25T19:11:28.933487+0000","last_active":"2024-04-25T19:11:37.171502+0000","last_peered":"2024-04-25T19:11:37.171502+0000","last_clean":"2024-04-25T19:11:37.171502+0000","last_became_active":"2024-04-25T19:11:28.933112+0000","last_became_peered":"2024-04-25T19:11:28.933112+0000","last_unstale":"2024-04-25T19:11:37.171502+0000","last_undegraded":"2024-04-25T19:11:37.171502+0000","last_fullsized":"2024-04-25T19:11:37.171502+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:27.901744+0000","last_clean_scrub_stamp":"2024-04-25T19:11:27.901744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:11:44.780178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990049+0000","last_change":"2024-04-25T19:11:29.927210+0000","last_active":"2024-04-25T19:11:36.990049+0000","last_peered":"2024-04-25T19:11:36.990049+0000","last_clean":"2024-04-25T19:11:36.990049+0000","last_became_active":"2024-04-25T19:11:29.926143+0000","last_became_peered":"2024-04-25T19:11:29.926143+0000","last_unstale":"2024-04-25T19:11:36.990049+0000","last_undegraded":"2024-04-25T19:11:36.990049+0000","last_fullsized":"2024-04-25T19:11:36.990049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:31.906041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171522+0000","last_change":"2024-04-25T19:11:33.974927+0000","last_active":"2024-04-25T19:11:37.171522+0000","last_peered":"2024-04-25T19:11:37.171522+0000","last_clean":"2024-04-25T19:11:37.171522+0000","last_became_active":"2024-04-25T19:11:33.974526+0000","last_became_peered":"2024-04-25T19:11:33.974526+0000","last_unstale":"2024-04-25T19:11:37.171522+0000","last_undegraded":"2024-04-25T19:11:37.171522+0000","last_fullsized":"2024-04-25T19:11:37.171522+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:32.941059+0000","last_clean_scrub_stamp":"2024-04-25T19:11:32.941059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:22:32.367757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961363+0000","last_change":"2024-04-25T19:11:29.943955+0000","last_active":"2024-04-25T19:11:34.961363+0000","last_peered":"2024-04-25T19:11:34.961363+0000","last_clean":"2024-04-25T19:11:34.961363+0000","last_became_active":"2024-04-25T19:11:29.943620+0000","last_became_peered":"2024-04-25T19:11:29.943620+0000","last_unstale":"2024-04-25T19:11:34.961363+0000","last_undegraded":"2024-04-25T19:11:34.961363+0000","last_fullsized":"2024-04-25T19:11:34.961363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960143+0000","last_change":"2024-04-25T19:11:29.944224+0000","last_active":"2024-04-25T19:11:34.960143+0000","last_peered":"2024-04-25T19:11:34.960143+0000","last_clean":"2024-04-25T19:11:34.960143+0000","last_became_active":"2024-04-25T19:11:29.943851+0000","last_became_peered":"2024-04-25T19:11:29.943851+0000","last_unstale":"2024-04-25T19:11:34.960143+0000","last_undegraded":"2024-04-25T19:11:34.960143+0000","last_fullsized":"2024-04-25T19:11:34.960143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:00:31.906041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963271+0000","last_change":"2024-04-25T19:11:29.948559+0000","last_active":"2024-04-25T19:11:34.963271+0000","last_peered":"2024-04-25T19:11:34.963271+0000","last_clean":"2024-04-25T19:11:34.963271+0000","last_became_active":"2024-04-25T19:11:29.948261+0000","last_became_peered":"2024-04-25T19:11:29.948261+0000","last_unstale":"2024-04-25T19:11:34.963271+0000","last_undegraded":"2024-04-25T19:11:34.963271+0000","last_fullsized":"2024-04-25T19:11:34.963271+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878297+0000","last_change":"2024-04-25T19:11:29.941924+0000","last_active":"2024-04-25T19:11:35.878297+0000","last_peered":"2024-04-25T19:11:35.878297+0000","last_clean":"2024-04-25T19:11:35.878297+0000","last_became_active":"2024-04-25T19:11:29.939317+0000","last_became_peered":"2024-04-25T19:11:29.939317+0000","last_unstale":"2024-04-25T19:11:35.878297+0000","last_undegraded":"2024-04-25T19:11:35.878297+0000","last_fullsized":"2024-04-25T19:11:35.878297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171676+0000","last_change":"2024-04-25T19:11:29.937716+0000","last_active":"2024-04-25T19:11:37.171676+0000","last_peered":"2024-04-25T19:11:37.171676+0000","last_clean":"2024-04-25T19:11:37.171676+0000","last_became_active":"2024-04-25T19:11:29.936902+0000","last_became_peered":"2024-04-25T19:11:29.936902+0000","last_unstale":"2024-04-25T19:11:37.171676+0000","last_undegraded":"2024-04-25T19:11:37.171676+0000","last_fullsized":"2024-04-25T19:11:37.171676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470475+0000","last_change":"2024-04-25T19:11:29.933948+0000","last_active":"2024-04-25T19:11:35.470475+0000","last_peered":"2024-04-25T19:11:35.470475+0000","last_clean":"2024-04-25T19:11:35.470475+0000","last_became_active":"2024-04-25T19:11:29.933609+0000","last_became_peered":"2024-04-25T19:11:29.933609+0000","last_unstale":"2024-04-25T19:11:35.470475+0000","last_undegraded":"2024-04-25T19:11:35.470475+0000","last_fullsized":"2024-04-25T19:11:35.470475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963143+0000","last_change":"2024-04-25T19:11:29.949696+0000","last_active":"2024-04-25T19:11:34.963143+0000","last_peered":"2024-04-25T19:11:34.963143+0000","last_clean":"2024-04-25T19:11:34.963143+0000","last_became_active":"2024-04-25T19:11:29.949244+0000","last_became_peered":"2024-04-25T19:11:29.949244+0000","last_unstale":"2024-04-25T19:11:34.963143+0000","last_undegraded":"2024-04-25T19:11:34.963143+0000","last_fullsized":"2024-04-25T19:11:34.963143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171872+0000","last_change":"2024-04-25T19:11:29.936507+0000","last_active":"2024-04-25T19:11:37.171872+0000","last_peered":"2024-04-25T19:11:37.171872+0000","last_clean":"2024-04-25T19:11:37.171872+0000","last_became_active":"2024-04-25T19:11:29.936194+0000","last_became_peered":"2024-04-25T19:11:29.936194+0000","last_unstale":"2024-04-25T19:11:37.171872+0000","last_undegraded":"2024-04-25T19:11:37.171872+0000","last_fullsized":"2024-04-25T19:11:37.171872+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961467+0000","last_change":"2024-04-25T19:11:29.949296+0000","last_active":"2024-04-25T19:11:34.961467+0000","last_peered":"2024-04-25T19:11:34.961467+0000","last_clean":"2024-04-25T19:11:34.961467+0000","last_became_active":"2024-04-25T19:11:29.948939+0000","last_became_peered":"2024-04-25T19:11:29.948939+0000","last_unstale":"2024-04-25T19:11:34.961467+0000","last_undegraded":"2024-04-25T19:11:34.961467+0000","last_fullsized":"2024-04-25T19:11:34.961467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:39.883364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171201+0000","last_change":"2024-04-25T19:11:29.937245+0000","last_active":"2024-04-25T19:11:37.171201+0000","last_peered":"2024-04-25T19:11:37.171201+0000","last_clean":"2024-04-25T19:11:37.171201+0000","last_became_active":"2024-04-25T19:11:29.936924+0000","last_became_peered":"2024-04-25T19:11:29.936924+0000","last_unstale":"2024-04-25T19:11:37.171201+0000","last_undegraded":"2024-04-25T19:11:37.171201+0000","last_fullsized":"2024-04-25T19:11:37.171201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:30:09.709588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963148+0000","last_change":"2024-04-25T19:11:29.939024+0000","last_active":"2024-04-25T19:11:34.963148+0000","last_peered":"2024-04-25T19:11:34.963148+0000","last_clean":"2024-04-25T19:11:34.963148+0000","last_became_active":"2024-04-25T19:11:29.938702+0000","last_became_peered":"2024-04-25T19:11:29.938702+0000","last_unstale":"2024-04-25T19:11:34.963148+0000","last_undegraded":"2024-04-25T19:11:34.963148+0000","last_fullsized":"2024-04-25T19:11:34.963148+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878155+0000","last_change":"2024-04-25T19:11:29.942883+0000","last_active":"2024-04-25T19:11:35.878155+0000","last_peered":"2024-04-25T19:11:35.878155+0000","last_clean":"2024-04-25T19:11:35.878155+0000","last_became_active":"2024-04-25T19:11:29.942371+0000","last_became_peered":"2024-04-25T19:11:29.942371+0000","last_unstale":"2024-04-25T19:11:35.878155+0000","last_undegraded":"2024-04-25T19:11:35.878155+0000","last_fullsized":"2024-04-25T19:11:35.878155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962242+0000","last_change":"2024-04-25T19:11:29.940287+0000","last_active":"2024-04-25T19:11:34.962242+0000","last_peered":"2024-04-25T19:11:34.962242+0000","last_clean":"2024-04-25T19:11:34.962242+0000","last_became_active":"2024-04-25T19:11:29.939973+0000","last_became_peered":"2024-04-25T19:11:29.939973+0000","last_unstale":"2024-04-25T19:11:34.962242+0000","last_undegraded":"2024-04-25T19:11:34.962242+0000","last_fullsized":"2024-04-25T19:11:34.962242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172189+0000","last_change":"2024-04-25T19:11:29.932331+0000","last_active":"2024-04-25T19:11:37.172189+0000","last_peered":"2024-04-25T19:11:37.172189+0000","last_clean":"2024-04-25T19:11:37.172189+0000","last_became_active":"2024-04-25T19:11:29.932030+0000","last_became_peered":"2024-04-25T19:11:29.932030+0000","last_unstale":"2024-04-25T19:11:37.172189+0000","last_undegraded":"2024-04-25T19:11:37.172189+0000","last_fullsized":"2024-04-25T19:11:37.172189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:05:48.917939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.960501+0000","last_change":"2024-04-25T19:11:29.930099+0000","last_active":"2024-04-25T19:11:34.960501+0000","last_peered":"2024-04-25T19:11:34.960501+0000","last_clean":"2024-04-25T19:11:34.960501+0000","last_became_active":"2024-04-25T19:11:29.929729+0000","last_became_peered":"2024-04-25T19:11:29.929729+0000","last_unstale":"2024-04-25T19:11:34.960501+0000","last_undegraded":"2024-04-25T19:11:34.960501+0000","last_fullsized":"2024-04-25T19:11:34.960501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:43:39.242588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963790+0000","last_change":"2024-04-25T19:11:29.940716+0000","last_active":"2024-04-25T19:11:34.963790+0000","last_peered":"2024-04-25T19:11:34.963790+0000","last_clean":"2024-04-25T19:11:34.963790+0000","last_became_active":"2024-04-25T19:11:29.940412+0000","last_became_peered":"2024-04-25T19:11:29.940412+0000","last_unstale":"2024-04-25T19:11:34.963790+0000","last_undegraded":"2024-04-25T19:11:34.963790+0000","last_fullsized":"2024-04-25T19:11:34.963790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:35.993870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.961354+0000","last_change":"2024-04-25T19:11:29.945896+0000","last_active":"2024-04-25T19:11:34.961354+0000","last_peered":"2024-04-25T19:11:34.961354+0000","last_clean":"2024-04-25T19:11:34.961354+0000","last_became_active":"2024-04-25T19:11:29.945590+0000","last_became_peered":"2024-04-25T19:11:29.945590+0000","last_unstale":"2024-04-25T19:11:34.961354+0000","last_undegraded":"2024-04-25T19:11:34.961354+0000","last_fullsized":"2024-04-25T19:11:34.961354+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:42.722556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470247+0000","last_change":"2024-04-25T19:11:29.939162+0000","last_active":"2024-04-25T19:11:35.470247+0000","last_peered":"2024-04-25T19:11:35.470247+0000","last_clean":"2024-04-25T19:11:35.470247+0000","last_became_active":"2024-04-25T19:11:29.938821+0000","last_became_peered":"2024-04-25T19:11:29.938821+0000","last_unstale":"2024-04-25T19:11:35.470247+0000","last_undegraded":"2024-04-25T19:11:35.470247+0000","last_fullsized":"2024-04-25T19:11:35.470247+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:52:14.593811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878300+0000","last_change":"2024-04-25T19:11:29.938034+0000","last_active":"2024-04-25T19:11:35.878300+0000","last_peered":"2024-04-25T19:11:35.878300+0000","last_clean":"2024-04-25T19:11:35.878300+0000","last_became_active":"2024-04-25T19:11:29.936771+0000","last_became_peered":"2024-04-25T19:11:29.936771+0000","last_unstale":"2024-04-25T19:11:35.878300+0000","last_undegraded":"2024-04-25T19:11:35.878300+0000","last_fullsized":"2024-04-25T19:11:35.878300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:23.833133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171095+0000","last_change":"2024-04-25T19:11:29.944382+0000","last_active":"2024-04-25T19:11:37.171095+0000","last_peered":"2024-04-25T19:11:37.171095+0000","last_clean":"2024-04-25T19:11:37.171095+0000","last_became_active":"2024-04-25T19:11:29.938702+0000","last_became_peered":"2024-04-25T19:11:29.938702+0000","last_unstale":"2024-04-25T19:11:37.171095+0000","last_undegraded":"2024-04-25T19:11:37.171095+0000","last_fullsized":"2024-04-25T19:11:37.171095+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:00.008382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:36.990488+0000","last_change":"2024-04-25T19:11:29.934918+0000","last_active":"2024-04-25T19:11:36.990488+0000","last_peered":"2024-04-25T19:11:36.990488+0000","last_clean":"2024-04-25T19:11:36.990488+0000","last_became_active":"2024-04-25T19:11:29.934600+0000","last_became_peered":"2024-04-25T19:11:29.934600+0000","last_unstale":"2024-04-25T19:11:36.990488+0000","last_undegraded":"2024-04-25T19:11:36.990488+0000","last_fullsized":"2024-04-25T19:11:36.990488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:29:15.027554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962499+0000","last_change":"2024-04-25T19:11:29.944854+0000","last_active":"2024-04-25T19:11:34.962499+0000","last_peered":"2024-04-25T19:11:34.962499+0000","last_clean":"2024-04-25T19:11:34.962499+0000","last_became_active":"2024-04-25T19:11:29.944528+0000","last_became_peered":"2024-04-25T19:11:29.944528+0000","last_unstale":"2024-04-25T19:11:34.962499+0000","last_undegraded":"2024-04-25T19:11:34.962499+0000","last_fullsized":"2024-04-25T19:11:34.962499+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:58:10.982626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963024+0000","last_change":"2024-04-25T19:11:29.944061+0000","last_active":"2024-04-25T19:11:34.963024+0000","last_peered":"2024-04-25T19:11:34.963024+0000","last_clean":"2024-04-25T19:11:34.963024+0000","last_became_active":"2024-04-25T19:11:29.943745+0000","last_became_peered":"2024-04-25T19:11:29.943745+0000","last_unstale":"2024-04-25T19:11:34.963024+0000","last_undegraded":"2024-04-25T19:11:34.963024+0000","last_fullsized":"2024-04-25T19:11:34.963024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:19.714879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172433+0000","last_change":"2024-04-25T19:11:29.944585+0000","last_active":"2024-04-25T19:11:37.172433+0000","last_peered":"2024-04-25T19:11:37.172433+0000","last_clean":"2024-04-25T19:11:37.172433+0000","last_became_active":"2024-04-25T19:11:29.944112+0000","last_became_peered":"2024-04-25T19:11:29.944112+0000","last_unstale":"2024-04-25T19:11:37.172433+0000","last_undegraded":"2024-04-25T19:11:37.172433+0000","last_fullsized":"2024-04-25T19:11:37.172433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:31.654149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.170915+0000","last_change":"2024-04-25T19:11:29.949991+0000","last_active":"2024-04-25T19:11:37.170915+0000","last_peered":"2024-04-25T19:11:37.170915+0000","last_clean":"2024-04-25T19:11:37.170915+0000","last_became_active":"2024-04-25T19:11:29.948567+0000","last_became_peered":"2024-04-25T19:11:29.948567+0000","last_unstale":"2024-04-25T19:11:37.170915+0000","last_undegraded":"2024-04-25T19:11:37.170915+0000","last_fullsized":"2024-04-25T19:11:37.170915+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:31:25.543650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171414+0000","last_change":"2024-04-25T19:11:29.930321+0000","last_active":"2024-04-25T19:11:37.171414+0000","last_peered":"2024-04-25T19:11:37.171414+0000","last_clean":"2024-04-25T19:11:37.171414+0000","last_became_active":"2024-04-25T19:11:29.929930+0000","last_became_peered":"2024-04-25T19:11:29.929930+0000","last_unstale":"2024-04-25T19:11:37.171414+0000","last_undegraded":"2024-04-25T19:11:37.171414+0000","last_fullsized":"2024-04-25T19:11:37.171414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.963030+0000","last_change":"2024-04-25T19:11:29.944694+0000","last_active":"2024-04-25T19:11:34.963030+0000","last_peered":"2024-04-25T19:11:34.963030+0000","last_clean":"2024-04-25T19:11:34.963030+0000","last_became_active":"2024-04-25T19:11:29.944055+0000","last_became_peered":"2024-04-25T19:11:29.944055+0000","last_unstale":"2024-04-25T19:11:34.963030+0000","last_undegraded":"2024-04-25T19:11:34.963030+0000","last_fullsized":"2024-04-25T19:11:34.963030+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:14:19.894645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.171775+0000","last_change":"2024-04-25T19:11:29.931815+0000","last_active":"2024-04-25T19:11:37.171775+0000","last_peered":"2024-04-25T19:11:37.171775+0000","last_clean":"2024-04-25T19:11:37.171775+0000","last_became_active":"2024-04-25T19:11:29.931523+0000","last_became_peered":"2024-04-25T19:11:29.931523+0000","last_unstale":"2024-04-25T19:11:37.171775+0000","last_undegraded":"2024-04-25T19:11:37.171775+0000","last_fullsized":"2024-04-25T19:11:37.171775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:43:42.722556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:34.962270+0000","last_change":"2024-04-25T19:11:29.951038+0000","last_active":"2024-04-25T19:11:34.962270+0000","last_peered":"2024-04-25T19:11:34.962270+0000","last_clean":"2024-04-25T19:11:34.962270+0000","last_became_active":"2024-04-25T19:11:29.950615+0000","last_became_peered":"2024-04-25T19:11:29.950615+0000","last_unstale":"2024-04-25T19:11:34.962270+0000","last_undegraded":"2024-04-25T19:11:34.962270+0000","last_fullsized":"2024-04-25T19:11:34.962270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:31.654149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.878604+0000","last_change":"2024-04-25T19:11:29.932658+0000","last_active":"2024-04-25T19:11:35.878604+0000","last_peered":"2024-04-25T19:11:35.878604+0000","last_clean":"2024-04-25T19:11:35.878604+0000","last_became_active":"2024-04-25T19:11:29.932270+0000","last_became_peered":"2024-04-25T19:11:29.932270+0000","last_unstale":"2024-04-25T19:11:35.878604+0000","last_undegraded":"2024-04-25T19:11:35.878604+0000","last_fullsized":"2024-04-25T19:11:35.878604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:12:11.271060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:35.470928+0000","last_change":"2024-04-25T19:11:29.929765+0000","last_active":"2024-04-25T19:11:35.470928+0000","last_peered":"2024-04-25T19:11:35.470928+0000","last_clean":"2024-04-25T19:11:35.470928+0000","last_became_active":"2024-04-25T19:11:29.929452+0000","last_became_peered":"2024-04-25T19:11:29.929452+0000","last_unstale":"2024-04-25T19:11:35.470928+0000","last_undegraded":"2024-04-25T19:11:35.470928+0000","last_fullsized":"2024-04-25T19:11:35.470928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:49:50.160790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T19:11:37.172357+0000","last_change":"2024-04-25T19:11:29.936916+0000","last_active":"2024-04-25T19:11:37.172357+0000","last_peered":"2024-04-25T19:11:37.172357+0000","last_clean":"2024-04-25T19:11:37.172357+0000","last_became_active":"2024-04-25T19:11:29.935057+0000","last_became_peered":"2024-04-25T19:11:29.935057+0000","last_unstale":"2024-04-25T19:11:37.172357+0000","last_undegraded":"2024-04-25T19:11:37.172357+0000","last_fullsized":"2024-04-25T19:11:37.172357+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:28.907723+0000","last_clean_scrub_stamp":"2024-04-25T19:11:28.907723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:04:35.993870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32012,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339828,"statfs":{"total":96636764160,"available":96603983872,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31440,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340400,"statfs":{"total":96636764160,"available":96604569600,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32020,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339820,"statfs":{"total":96636764160,"available":96603975680,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:11:56.407 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T19:11:56.407 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T19:11:56.407 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T19:11:56.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T19:11:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.551+0000 7f8750ed1640 1 Processor -- start 2024-04-25T19:11:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.551+0000 7f8750ed1640 1 -- start start 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.552+0000 7f8750ed1640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f874c160180 0x7f874c163c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.552+0000 7f8750ed1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.552+0000 7f8750ed1640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f874c164bb0 con 0x7f874c0d06c0 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.552+0000 7f8750ed1640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f874c0d4510 con 0x7f874c160180 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.552+0000 7f8750ed1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f874c0d2ea0 con 0x7f874c0d2550 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.552+0000 7f874a59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f874c160180 0x7f874c163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.552+0000 7f874a59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f874c160180 0x7f874c163c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37396/0 (socket says 172.21.15.28:37396) 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.552+0000 7f874a59c640 1 -- 172.21.15.28:0/1622705019 learned_addr learned my addr 172.21.15.28:0/1622705019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.553+0000 7f874ad9d640 1 --2- 172.21.15.28:0/1622705019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.553+0000 7f874a59c640 1 -- 172.21.15.28:0/1622705019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 msgr2=0x7f874c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:56.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.553+0000 7f874a59c640 1 --2- 172.21.15.28:0/1622705019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.553+0000 7f874a59c640 1 -- 172.21.15.28:0/1622705019 >> v1:172.21.15.53:6789/0 conn(0x7f874c0d06c0 legacy=0x7f874c0b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.553+0000 7f874a59c640 1 -- 172.21.15.28:0/1622705019 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874c164310 con 0x7f874c160180 2024-04-25T19:11:56.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.553+0000 7f874ad9d640 1 --2- 172.21.15.28:0/1622705019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.553+0000 7f874a59c640 1 --2- 172.21.15.28:0/1622705019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c160180 0x7f874c163c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f874c0d4690 tx=0x7f873c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cbe752dc536c9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8749d9b640 1 -- 172.21.15.28:0/1622705019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c037070 con 0x7f874c160180 2024-04-25T19:11:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8749d9b640 1 -- 172.21.15.28:0/1622705019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f873c032440 con 0x7f874c160180 2024-04-25T19:11:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8749d9b640 1 -- 172.21.15.28:0/1622705019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c032810 con 0x7f874c160180 2024-04-25T19:11:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8750ed1640 1 -- 172.21.15.28:0/1622705019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c160180 msgr2=0x7f874c163c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1622705019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c160180 0x7f874c163c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f874c0d4690 tx=0x7f873c02a250 comp rx=0 tx=0).stop 2024-04-25T19:11:56.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8750ed1640 1 -- 172.21.15.28:0/1622705019 shutdown_connections 2024-04-25T19:11:56.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1622705019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1622705019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c160180 0x7f874c163c80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8750ed1640 1 -- 172.21.15.28:0/1622705019 >> 172.21.15.28:0/1622705019 conn(0x7f874c06c750 msgr2=0x7f874c06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:56.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8750ed1640 1 -- 172.21.15.28:0/1622705019 shutdown_connections 2024-04-25T19:11:56.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.554+0000 7f8750ed1640 1 -- 172.21.15.28:0/1622705019 wait complete. 2024-04-25T19:11:56.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.556+0000 7f8750ed1640 1 Processor -- start 2024-04-25T19:11:56.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.556+0000 7f8750ed1640 1 -- start start 2024-04-25T19:11:56.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f8750ed1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c0d06c0 0x7f874c160590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f8750ed1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f8750ed1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f874c160180 0x7f874c146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f8750ed1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f874c0d0ad0 con 0x7f874c0d06c0 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f8750ed1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f874c1648d0 con 0x7f874c160180 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f8750ed1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f874c0d49a0 con 0x7f874c0d2550 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c0d06c0 0x7f874c160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c0d06c0 0x7f874c160590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37412/0 (socket says 172.21.15.28:37412) 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 -- 172.21.15.28:0/1902935366 learned_addr learned my addr 172.21.15.28:0/1902935366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874ad9d640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f874c160180 0x7f874c146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f8743fff640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 -- 172.21.15.28:0/1902935366 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 msgr2=0x7f874c146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c146670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 -- 172.21.15.28:0/1902935366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f874c160180 msgr2=0x7f874c146bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f874c160180 0x7f874c146bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 -- 172.21.15.28:0/1902935366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f874c160ad0 con 0x7f874c0d06c0 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.558+0000 7f874a59c640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c0d06c0 0x7f874c160590 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f873c02a290 tx=0x7f873c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.559+0000 7f8742ffd640 1 -- 172.21.15.28:0/1902935366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c037040 con 0x7f874c0d06c0 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.559+0000 7f8742ffd640 1 -- 172.21.15.28:0/1902935366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f873c042070 con 0x7f874c0d06c0 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.559+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f874c162df0 con 0x7f874c0d06c0 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.559+0000 7f8742ffd640 1 -- 172.21.15.28:0/1902935366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f873c030bb0 con 0x7f874c0d06c0 2024-04-25T19:11:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.559+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f874c1633b0 con 0x7f874c0d06c0 2024-04-25T19:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.560+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f874c050740 con 0x7f874c0d06c0 2024-04-25T19:11:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.560+0000 7f8742ffd640 1 -- 172.21.15.28:0/1902935366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f873c047460 con 0x7f874c0d06c0 2024-04-25T19:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.561+0000 7f8742ffd640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f872c0744b0 0x7f872c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.561+0000 7f8742ffd640 1 -- 172.21.15.28:0/1902935366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f873c0b60b0 con 0x7f874c0d06c0 2024-04-25T19:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.561+0000 7f8743fff640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f872c0744b0 0x7f872c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.561+0000 7f8743fff640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f872c0744b0 0x7f872c0769c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f874c157f00 tx=0x7f8734009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.565+0000 7f8742ffd640 1 -- 172.21.15.28:0/1902935366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f873c07feb0 con 0x7f874c0d06c0 2024-04-25T19:11:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.789+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f874c0d3f90 con 0x7f874c0d06c0 2024-04-25T19:11:56.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.790+0000 7f8742ffd640 1 -- 172.21.15.28:0/1902935366 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f873c0841b0 con 0x7f874c0d06c0 2024-04-25T19:11:56.792 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:56.792 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f872c0744b0 msgr2=0x7f872c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f872c0744b0 0x7f872c0769c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f874c157f00 tx=0x7f8734009040 comp rx=0 tx=0).stop 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c0d06c0 msgr2=0x7f874c160590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c0d06c0 0x7f874c160590 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f873c02a290 tx=0x7f873c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 shutdown_connections 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f872c0744b0 0x7f872c0769c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f874c160180 0x7f874c146bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f874c0d2550 0x7f874c146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 --2- 172.21.15.28:0/1902935366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f874c0d06c0 0x7f874c160590 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 >> 172.21.15.28:0/1902935366 conn(0x7f874c06c750 msgr2=0x7f874c0b6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 shutdown_connections 2024-04-25T19:11:56.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.792+0000 7f8750ed1640 1 -- 172.21.15.28:0/1902935366 wait complete. 2024-04-25T19:11:56.805 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T19:11:56.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:56.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.915+0000 7f7e7f405640 1 Processor -- start 2024-04-25T19:11:56.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.915+0000 7f7e7f405640 1 -- start start 2024-04-25T19:11:56.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7f405640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 0x7f7e78163c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7f405640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e780d06c0 0x7f7e780b9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7f405640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e78164bb0 con 0x7f7e780d2550 2024-04-25T19:11:56.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7f405640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f7e780d4510 con 0x7f7e780d06c0 2024-04-25T19:11:56.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7f405640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7e78164310 con 0x7f7e78160180 2024-04-25T19:11:56.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7c9a0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e780d06c0 0x7f7e780b9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7c9a0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e780d06c0 0x7f7e780b9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37416/0 (socket says 172.21.15.28:37416) 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7c9a0640 1 -- 172.21.15.28:0/510168889 learned_addr learned my addr 172.21.15.28:0/510168889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7d9a2640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e780d2550 0x7f7e780d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59026/0 (socket says 172.21.15.28:59026) 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.916+0000 7f7e7d1a1640 1 --2- 172.21.15.28:0/510168889 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 0x7f7e78163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.917+0000 7f7e7c9a0640 1 -- 172.21.15.28:0/510168889 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 msgr2=0x7f7e78163c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.917+0000 7f7e7c9a0640 1 --2- 172.21.15.28:0/510168889 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 0x7f7e78163c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.917+0000 7f7e7c9a0640 1 -- 172.21.15.28:0/510168889 >> v1:172.21.15.28:6789/0 conn(0x7f7e780d2550 legacy=0x7f7e780d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.917+0000 7f7e7c9a0640 1 -- 172.21.15.28:0/510168889 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e781645f0 con 0x7f7e780d06c0 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.917+0000 7f7e6ffff640 1 -- 172.21.15.28:0/510168889 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3783216600 0 0) 0x7f7e78164bb0 con 0x7f7e780d2550 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.917+0000 7f7e7c9a0640 1 --2- 172.21.15.28:0/510168889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d06c0 0x7f7e780b9090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009870 tx=0x7f7e6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e5241ae3b90372 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e6ffff640 1 -- 172.21.15.28:0/510168889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68037070 con 0x7f7e780d06c0 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e6ffff640 1 -- 172.21.15.28:0/510168889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e68032440 con 0x7f7e780d06c0 2024-04-25T19:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e6ffff640 1 -- 172.21.15.28:0/510168889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68032810 con 0x7f7e780d06c0 2024-04-25T19:11:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e7f405640 1 -- 172.21.15.28:0/510168889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d06c0 msgr2=0x7f7e780b9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e7f405640 1 --2- 172.21.15.28:0/510168889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d06c0 0x7f7e780b9090 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009870 tx=0x7f7e6802a270 comp rx=0 tx=0).stop 2024-04-25T19:11:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e7f405640 1 -- 172.21.15.28:0/510168889 shutdown_connections 2024-04-25T19:11:56.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e7f405640 1 --2- 172.21.15.28:0/510168889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d06c0 0x7f7e780b9090 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e7f405640 1 --2- 172.21.15.28:0/510168889 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 0x7f7e78163c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e7f405640 1 -- 172.21.15.28:0/510168889 >> 172.21.15.28:0/510168889 conn(0x7f7e7806c750 msgr2=0x7f7e7806cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:56.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e7f405640 1 -- 172.21.15.28:0/510168889 shutdown_connections 2024-04-25T19:11:56.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.918+0000 7f7e7f405640 1 -- 172.21.15.28:0/510168889 wait complete. 2024-04-25T19:11:56.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.920+0000 7f7e7f405640 1 Processor -- start 2024-04-25T19:11:56.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.920+0000 7f7e7f405640 1 -- start start 2024-04-25T19:11:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.921+0000 7f7e7f405640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e780d06c0 0x7f7e781f71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.921+0000 7f7e7f405640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d2550 0x7f7e78159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.921+0000 7f7e7f405640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 0x7f7e781f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.921+0000 7f7e7f405640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e780b95d0 con 0x7f7e780d2550 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.921+0000 7f7e7f405640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7e780d0ad0 con 0x7f7e780d06c0 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.921+0000 7f7e7f405640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7e780b9750 con 0x7f7e78160180 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d2550 0x7f7e78159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d2550 0x7f7e78159d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37430/0 (socket says 172.21.15.28:37430) 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 -- 172.21.15.28:0/2803127749 learned_addr learned my addr 172.21.15.28:0/2803127749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7d1a1640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e780d06c0 0x7f7e781f71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7d9a2640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 0x7f7e781f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 -- 172.21.15.28:0/2803127749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 msgr2=0x7f7e781f5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 0x7f7e781f5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 -- 172.21.15.28:0/2803127749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e780d06c0 msgr2=0x7f7e781f71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e780d06c0 0x7f7e781f71a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 -- 172.21.15.28:0/2803127749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7815a290 con 0x7f7e780d2550 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7c9a0640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d2550 0x7f7e78159d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009870 tx=0x7f7e680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e6dffb640 1 -- 172.21.15.28:0/2803127749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68041020 con 0x7f7e780d2550 2024-04-25T19:11:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e6dffb640 1 -- 172.21.15.28:0/2803127749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e68030640 con 0x7f7e780d2550 2024-04-25T19:11:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e6dffb640 1 -- 172.21.15.28:0/2803127749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68037040 con 0x7f7e780d2550 2024-04-25T19:11:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e78159400 con 0x7f7e780d2550 2024-04-25T19:11:56.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.922+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e78159980 con 0x7f7e780d2550 2024-04-25T19:11:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.923+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e78068f50 con 0x7f7e780d2550 2024-04-25T19:11:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.924+0000 7f7e6dffb640 1 -- 172.21.15.28:0/2803127749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7e68044020 con 0x7f7e780d2550 2024-04-25T19:11:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.924+0000 7f7e6dffb640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e5c0744e0 0x7f7e5c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.925+0000 7f7e6dffb640 1 -- 172.21.15.28:0/2803127749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7e680b5d60 con 0x7f7e780d2550 2024-04-25T19:11:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.925+0000 7f7e7d1a1640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e5c0744e0 0x7f7e5c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.925+0000 7f7e7d1a1640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e5c0744e0 0x7f7e5c0769f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009970 tx=0x7f7e74008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:56.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:56.929+0000 7f7e6dffb640 1 -- 172.21.15.28:0/2803127749 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7e78068f50 con 0x7f7e780d2550 2024-04-25T19:11:57.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.119+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e78050740 con 0x7f7e780d2550 2024-04-25T19:11:57.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.120+0000 7f7e6dffb640 1 -- 172.21.15.28:0/2803127749 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f7e6804b030 con 0x7f7e780d2550 2024-04-25T19:11:57.122 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:57.122 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e5c0744e0 msgr2=0x7f7e5c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e5c0744e0 0x7f7e5c0769f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7e74009970 tx=0x7f7e74008040 comp rx=0 tx=0).stop 2024-04-25T19:11:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d2550 msgr2=0x7f7e78159d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d2550 0x7f7e78159d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7e68009870 tx=0x7f7e680046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 shutdown_connections 2024-04-25T19:11:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e5c0744e0 0x7f7e5c0769f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e78160180 0x7f7e781f5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e780d2550 0x7f7e78159d50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 --2- 172.21.15.28:0/2803127749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e780d06c0 0x7f7e781f71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 >> 172.21.15.28:0/2803127749 conn(0x7f7e7806c750 msgr2=0x7f7e7815dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 shutdown_connections 2024-04-25T19:11:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.122+0000 7f7e7f405640 1 -- 172.21.15.28:0/2803127749 wait complete. 2024-04-25T19:11:57.136 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:11:57.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:11:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.251+0000 7efc92242640 1 Processor -- start 2024-04-25T19:11:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.251+0000 7efc92242640 1 -- start start 2024-04-25T19:11:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.252+0000 7efc92242640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efc8c0b32d0 0x7efc8c0b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.252+0000 7efc92242640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c0b3c20 0x7efc8c0d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.252+0000 7efc92242640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc8c09d1a0 con 0x7efc8c09cd90 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.252+0000 7efc92242640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7efc8c06df50 con 0x7efc8c0b32d0 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.252+0000 7efc92242640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efc8c0d4870 con 0x7efc8c0b3c20 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.252+0000 7efc8b7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efc8c09cd90 0x7efc8c06e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59040/0 (socket says 172.21.15.28:59040) 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.252+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2390289050 learned_addr learned my addr 172.21.15.28:0/2390289050 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8affd640 1 --2- 172.21.15.28:0/2390289050 >> v2:172.21.15.53:3300/0 conn(0x7efc8c0b32d0 0x7efc8c0b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525340525 0 0) 0x7efc8c09d1a0 con 0x7efc8c09cd90 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc74003520 con 0x7efc8c09cd90 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c0b3c20 0x7efc8c0d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2136391635 0 0) 0x7efc74003520 con 0x7efc8c09cd90 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c0b3c20 msgr2=0x7efc8c0d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c0b3c20 0x7efc8c0d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 >> v2:172.21.15.53:3300/0 conn(0x7efc8c0b32d0 msgr2=0x7efc8c0b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 --2- 172.21.15.28:0/2390289050 >> v2:172.21.15.53:3300/0 conn(0x7efc8c0b32d0 0x7efc8c0b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c06d960 con 0x7efc8c09cd90 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7efc7c001a10 con 0x7efc8c09cd90 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 >> v1:172.21.15.28:6789/0 conn(0x7efc8c09cd90 legacy=0x7efc8c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc740039b0 0x7efc74003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c0b3c20 0x7efc74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc74004d40 0x7efc74005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c06df50 con 0x7efc74004d40 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c0d4870 con 0x7efc740039b0 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efc7400ba40 con 0x7efc8c0b3c20 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc74004d40 0x7efc74005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8b7fe640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc740039b0 0x7efc74003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8affd640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c0b3c20 0x7efc74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8bfff640 1 -- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c0b3c20 msgr2=0x7efc74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c0b3c20 0x7efc74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8bfff640 1 -- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc740039b0 msgr2=0x7efc74003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc740039b0 0x7efc74003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.253+0000 7efc8bfff640 1 -- 172.21.15.28:0/2390289050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc74003520 con 0x7efc74004d40 2024-04-25T19:11:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.254+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc74004d40 0x7efc74005150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0c7470 tx=0x7efc74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1480be85d06ab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.254+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84036070 con 0x7efc74004d40 2024-04-25T19:11:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.254+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc84031400 con 0x7efc74004d40 2024-04-25T19:11:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.254+0000 7efc8a7fc640 1 -- 172.21.15.28:0/2390289050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc84009690 con 0x7efc74004d40 2024-04-25T19:11:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.255+0000 7efc92242640 1 -- 172.21.15.28:0/2390289050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc74004d40 msgr2=0x7efc74005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.255+0000 7efc92242640 1 --2- 172.21.15.28:0/2390289050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc74004d40 0x7efc74005150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0c7470 tx=0x7efc74005830 comp rx=0 tx=0).stop 2024-04-25T19:11:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.255+0000 7efc8bfff640 1 -- 172.21.15.28:0/2390289050 reap_dead start 2024-04-25T19:11:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.255+0000 7efc92242640 1 -- 172.21.15.28:0/2390289050 shutdown_connections 2024-04-25T19:11:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.255+0000 7efc92242640 1 -- 172.21.15.28:0/2390289050 >> 172.21.15.28:0/2390289050 conn(0x7efc8c0d03f0 msgr2=0x7efc8c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.255+0000 7efc92242640 1 -- 172.21.15.28:0/2390289050 shutdown_connections 2024-04-25T19:11:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.255+0000 7efc92242640 1 -- 172.21.15.28:0/2390289050 wait complete. 2024-04-25T19:11:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.257+0000 7efc92242640 1 Processor -- start 2024-04-25T19:11:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.257+0000 7efc92242640 1 -- start start 2024-04-25T19:11:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.258+0000 7efc92242640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c0caa20 0x7efc8c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.258+0000 7efc92242640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc8c1f6d10 0x7efc8c09fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.258+0000 7efc92242640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c1db390 0x7efc8c1db7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.258+0000 7efc92242640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c0b9180 con 0x7efc8c0caa20 2024-04-25T19:11:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.258+0000 7efc92242640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efc8c06dc70 con 0x7efc8c1f6d10 2024-04-25T19:11:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.258+0000 7efc92242640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efc8c06ea10 con 0x7efc8c1db390 2024-04-25T19:11:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.258+0000 7efc8b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c0caa20 0x7efc8c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.258+0000 7efc8b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c0caa20 0x7efc8c0cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37450/0 (socket says 172.21.15.28:37450) 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2595140934 learned_addr learned my addr 172.21.15.28:0/2595140934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8affd640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc8c1f6d10 0x7efc8c09fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8bfff640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c1db390 0x7efc8c1db7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c1db390 msgr2=0x7efc8c1db7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8b7fe640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c1db390 0x7efc8c1db7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc8c1f6d10 msgr2=0x7efc8c09fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8b7fe640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc8c1f6d10 0x7efc8c09fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8b7fe640 1 -- 172.21.15.28:0/2595140934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c1dc230 con 0x7efc8c0caa20 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8affd640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc8c1f6d10 0x7efc8c09fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc8b7fe640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c0caa20 0x7efc8c0cbf80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00a3d0 tx=0x7efc7c00a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc73fff640 1 -- 172.21.15.28:0/2595140934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c017020 con 0x7efc8c0caa20 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc73fff640 1 -- 172.21.15.28:0/2595140934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc7c0136c0 con 0x7efc8c0caa20 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc8c1dbce0 con 0x7efc8c0caa20 2024-04-25T19:11:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc73fff640 1 -- 172.21.15.28:0/2595140934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c013ab0 con 0x7efc8c0caa20 2024-04-25T19:11:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.259+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc8c0cb4a0 con 0x7efc8c0caa20 2024-04-25T19:11:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.260+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc8c0b4030 con 0x7efc8c0caa20 2024-04-25T19:11:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.261+0000 7efc73fff640 1 -- 172.21.15.28:0/2595140934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efc7c01a020 con 0x7efc8c0caa20 2024-04-25T19:11:57.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.261+0000 7efc73fff640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc780744b0 0x7efc780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.262+0000 7efc8affd640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc780744b0 0x7efc780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.262+0000 7efc73fff640 1 -- 172.21.15.28:0/2595140934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efc7c0964a0 con 0x7efc8c0caa20 2024-04-25T19:11:57.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.262+0000 7efc8affd640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc780744b0 0x7efc780769c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0ca760 tx=0x7efc80008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.266+0000 7efc73fff640 1 -- 172.21.15.28:0/2595140934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc8c0b4030 con 0x7efc8c0caa20 2024-04-25T19:11:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.451+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efc8c050740 con 0x7efc8c0caa20 2024-04-25T19:11:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.452+0000 7efc73fff640 1 -- 172.21.15.28:0/2595140934 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7efc7c021020 con 0x7efc8c0caa20 2024-04-25T19:11:57.454 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:11:57.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc780744b0 msgr2=0x7efc780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc780744b0 0x7efc780769c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efc8c0ca760 tx=0x7efc80008040 comp rx=0 tx=0).stop 2024-04-25T19:11:57.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c0caa20 msgr2=0x7efc8c0cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c0caa20 0x7efc8c0cbf80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00a3d0 tx=0x7efc7c00a890 comp rx=0 tx=0).stop 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 shutdown_connections 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efc780744b0 0x7efc780769c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efc8c1db390 0x7efc8c1db7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc8c1f6d10 0x7efc8c09fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 --2- 172.21.15.28:0/2595140934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc8c0caa20 0x7efc8c0cbf80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 >> 172.21.15.28:0/2595140934 conn(0x7efc8c0d03f0 msgr2=0x7efc8c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 shutdown_connections 2024-04-25T19:11:57.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.454+0000 7efc92242640 1 -- 172.21.15.28:0/2595140934 wait complete. 2024-04-25T19:11:57.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:11:57.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.580+0000 7f85fd230640 1 Processor -- start 2024-04-25T19:11:57.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.580+0000 7f85fd230640 1 -- start start 2024-04-25T19:11:57.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.581+0000 7f85fd230640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f8070b60 0x7f85f815d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.581+0000 7f85fd230640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85f806fc30 0x7f85f8070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.581+0000 7f85fd230640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f85f8160190 0x7f85f81605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.581+0000 7f85fd230640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f85f815e0d0 con 0x7f85f806fc30 2024-04-25T19:11:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.581+0000 7f85fd230640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f85f8153b00 con 0x7f85f8160190 2024-04-25T19:11:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.581+0000 7f85fd230640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f85f8070580 con 0x7f85f8070b60 2024-04-25T19:11:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f659c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85f806fc30 0x7f85f8070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f659c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85f806fc30 0x7f85f8070040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37466/0 (socket says 172.21.15.28:37466) 2024-04-25T19:11:57.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f659c640 1 -- 172.21.15.28:0/629518687 learned_addr learned my addr 172.21.15.28:0/629518687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f759e640 1 --2- 172.21.15.28:0/629518687 >> v2:172.21.15.53:3300/0 conn(0x7f85f8160190 0x7f85f81605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f6d9d640 1 --2- 172.21.15.28:0/629518687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f8070b60 0x7f85f815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f659c640 1 -- 172.21.15.28:0/629518687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f8070b60 msgr2=0x7f85f815d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f659c640 1 --2- 172.21.15.28:0/629518687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f8070b60 0x7f85f815d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f659c640 1 -- 172.21.15.28:0/629518687 >> v2:172.21.15.53:3300/0 conn(0x7f85f8160190 msgr2=0x7f85f81605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f659c640 1 --2- 172.21.15.28:0/629518687 >> v2:172.21.15.53:3300/0 conn(0x7f85f8160190 0x7f85f81605a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f659c640 1 -- 172.21.15.28:0/629518687 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f80af3c0 con 0x7f85f806fc30 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.582+0000 7f85f759e640 1 --2- 172.21.15.28:0/629518687 >> v2:172.21.15.53:3300/0 conn(0x7f85f8160190 0x7f85f81605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85f659c640 1 --2- 172.21.15.28:0/629518687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f806fc30 0x7f85f8070040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f85f8070730 tx=0x7f85e002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4235fab142e454ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85f5d9b640 1 -- 172.21.15.28:0/629518687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0037070 con 0x7f85f806fc30 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85f5d9b640 1 -- 172.21.15.28:0/629518687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e0032440 con 0x7f85f806fc30 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85f5d9b640 1 -- 172.21.15.28:0/629518687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e0032810 con 0x7f85f806fc30 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 -- 172.21.15.28:0/629518687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f806fc30 msgr2=0x7f85f8070040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 --2- 172.21.15.28:0/629518687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f806fc30 0x7f85f8070040 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f85f8070730 tx=0x7f85e002a130 comp rx=0 tx=0).stop 2024-04-25T19:11:57.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 -- 172.21.15.28:0/629518687 shutdown_connections 2024-04-25T19:11:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 --2- 172.21.15.28:0/629518687 >> v2:172.21.15.53:3300/0 conn(0x7f85f8160190 0x7f85f81605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 --2- 172.21.15.28:0/629518687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f806fc30 0x7f85f8070040 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 --2- 172.21.15.28:0/629518687 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f8070b60 0x7f85f815d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 -- 172.21.15.28:0/629518687 >> 172.21.15.28:0/629518687 conn(0x7f85f80742c0 msgr2=0x7f85f806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 -- 172.21.15.28:0/629518687 shutdown_connections 2024-04-25T19:11:57.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.583+0000 7f85fd230640 1 -- 172.21.15.28:0/629518687 wait complete. 2024-04-25T19:11:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.585+0000 7f85fd230640 1 Processor -- start 2024-04-25T19:11:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.586+0000 7f85fd230640 1 -- start start 2024-04-25T19:11:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85fd230640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f806fc30 0x7f85f814dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85fd230640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85f8070b60 0x7f85f81f5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85fd230640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f8160190 0x7f85f80d0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85fd230640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f85f815dd60 con 0x7f85f8160190 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85fd230640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f85f80af980 con 0x7f85f8070b60 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85fd230640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f85f815d940 con 0x7f85f806fc30 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f8160190 0x7f85f80d0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f8160190 0x7f85f80d0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37472/0 (socket says 172.21.15.28:37472) 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 -- 172.21.15.28:0/1236354920 learned_addr learned my addr 172.21.15.28:0/1236354920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f6d9d640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f806fc30 0x7f85f814dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f659c640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85f8070b60 0x7f85f81f5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 -- 172.21.15.28:0/1236354920 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f806fc30 msgr2=0x7f85f814dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f806fc30 0x7f85f814dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 -- 172.21.15.28:0/1236354920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85f8070b60 msgr2=0x7f85f81f5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85f8070b60 0x7f85f81f5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 -- 172.21.15.28:0/1236354920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85f80ce490 con 0x7f85f8160190 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.587+0000 7f85f759e640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f8160190 0x7f85f80d0620 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f85e800cd10 tx=0x7f85e800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.588+0000 7f85ef7fe640 1 -- 172.21.15.28:0/1236354920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e8003b90 con 0x7f85f8160190 2024-04-25T19:11:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.588+0000 7f85ef7fe640 1 -- 172.21.15.28:0/1236354920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e8003d50 con 0x7f85f8160190 2024-04-25T19:11:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.588+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85f80ce730 con 0x7f85f8160190 2024-04-25T19:11:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.588+0000 7f85ef7fe640 1 -- 172.21.15.28:0/1236354920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e8004260 con 0x7f85f8160190 2024-04-25T19:11:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.588+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85f80cecf0 con 0x7f85f8160190 2024-04-25T19:11:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.589+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85f8050740 con 0x7f85f8160190 2024-04-25T19:11:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.590+0000 7f85ef7fe640 1 -- 172.21.15.28:0/1236354920 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f85e8004400 con 0x7f85f8160190 2024-04-25T19:11:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.590+0000 7f85ef7fe640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85d80744b0 0x7f85d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.590+0000 7f85ef7fe640 1 -- 172.21.15.28:0/1236354920 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f85e8095de0 con 0x7f85f8160190 2024-04-25T19:11:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.590+0000 7f85f6d9d640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85d80744b0 0x7f85d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.591+0000 7f85f6d9d640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85d80744b0 0x7f85d80769c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f85f000a9b0 tx=0x7f85f0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.595+0000 7f85ef7fe640 1 -- 172.21.15.28:0/1236354920 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f85e8014030 con 0x7f85f8160190 2024-04-25T19:11:57.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.780+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f85f80697c0 con 0x7f85f8160190 2024-04-25T19:11:57.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.781+0000 7f85ef7fe640 1 -- 172.21.15.28:0/1236354920 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f85e805fc70 con 0x7f85f8160190 2024-04-25T19:11:57.783 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:11:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85d80744b0 msgr2=0x7f85d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85d80744b0 0x7f85d80769c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f85f000a9b0 tx=0x7f85f0008040 comp rx=0 tx=0).stop 2024-04-25T19:11:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f8160190 msgr2=0x7f85f80d0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f8160190 0x7f85f80d0620 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f85e800cd10 tx=0x7f85e800a650 comp rx=0 tx=0).stop 2024-04-25T19:11:57.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 shutdown_connections 2024-04-25T19:11:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85d80744b0 0x7f85d80769c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f8160190 0x7f85f80d0620 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85f8070b60 0x7f85f81f5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 --2- 172.21.15.28:0/1236354920 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85f806fc30 0x7f85f814dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 >> 172.21.15.28:0/1236354920 conn(0x7f85f80742c0 msgr2=0x7f85f80b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 shutdown_connections 2024-04-25T19:11:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.783+0000 7f85fd230640 1 -- 172.21.15.28:0/1236354920 wait complete. 2024-04-25T19:11:57.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:11:57.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.910+0000 7f3d4184a640 1 Processor -- start 2024-04-25T19:11:57.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.910+0000 7f3d4184a640 1 -- start start 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.911+0000 7f3d4184a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d3c0b21c0 0x7f3d3c15dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.911+0000 7f3d4184a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c161230 0x7f3d3c06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.911+0000 7f3d4184a640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3c0b29a0 con 0x7f3d3c0d1f80 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.911+0000 7f3d4184a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f3d3c06c570 con 0x7f3d3c0b21c0 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.911+0000 7f3d4184a640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c0d2540 con 0x7f3d3c161230 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.911+0000 7f3d3affd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3d3c0d1f80 0x7f3d3c0b1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59044/0 (socket says 172.21.15.28:59044) 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.911+0000 7f3d3affd640 1 -- 172.21.15.28:0/584229538 learned_addr learned my addr 172.21.15.28:0/584229538 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.911+0000 7f3d3a7fc640 1 --2- 172.21.15.28:0/584229538 >> v2:172.21.15.28:3300/0 conn(0x7f3d3c0b21c0 0x7f3d3c15dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.912+0000 7f3d3b7fe640 1 --2- 172.21.15.28:0/584229538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c161230 0x7f3d3c06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.912+0000 7f3d3a7fc640 1 -- 172.21.15.28:0/584229538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c161230 msgr2=0x7f3d3c06ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:57.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.912+0000 7f3d3a7fc640 1 --2- 172.21.15.28:0/584229538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c161230 0x7f3d3c06ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.912+0000 7f3d3a7fc640 1 -- 172.21.15.28:0/584229538 >> v1:172.21.15.28:6789/0 conn(0x7f3d3c0d1f80 legacy=0x7f3d3c0b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.912+0000 7f3d39ffb640 1 -- 172.21.15.28:0/584229538 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3247372454 0 0) 0x7f3d3c0b29a0 con 0x7f3d3c0d1f80 2024-04-25T19:11:57.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.912+0000 7f3d3a7fc640 1 -- 172.21.15.28:0/584229538 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c0d2820 con 0x7f3d3c0b21c0 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d3a7fc640 1 --2- 172.21.15.28:0/584229538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c0b21c0 0x7f3d3c15dd70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c06c6f0 tx=0x7f3d3002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c57e099097af64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d39ffb640 1 -- 172.21.15.28:0/584229538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30037070 con 0x7f3d3c0b21c0 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d39ffb640 1 -- 172.21.15.28:0/584229538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3002ae50 con 0x7f3d3c0b21c0 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d39ffb640 1 -- 172.21.15.28:0/584229538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30032660 con 0x7f3d3c0b21c0 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d4184a640 1 -- 172.21.15.28:0/584229538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c0b21c0 msgr2=0x7f3d3c15dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d4184a640 1 --2- 172.21.15.28:0/584229538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c0b21c0 0x7f3d3c15dd70 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c06c6f0 tx=0x7f3d3002a100 comp rx=0 tx=0).stop 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d4184a640 1 -- 172.21.15.28:0/584229538 shutdown_connections 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d4184a640 1 --2- 172.21.15.28:0/584229538 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c161230 0x7f3d3c06ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d4184a640 1 --2- 172.21.15.28:0/584229538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c0b21c0 0x7f3d3c15dd70 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.913+0000 7f3d4184a640 1 -- 172.21.15.28:0/584229538 >> 172.21.15.28:0/584229538 conn(0x7f3d3c073d30 msgr2=0x7f3d3c09b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.914+0000 7f3d4184a640 1 -- 172.21.15.28:0/584229538 shutdown_connections 2024-04-25T19:11:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.914+0000 7f3d4184a640 1 -- 172.21.15.28:0/584229538 wait complete. 2024-04-25T19:11:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.915+0000 7f3d4184a640 1 Processor -- start 2024-04-25T19:11:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.916+0000 7f3d4184a640 1 -- start start 2024-04-25T19:11:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d4184a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d3c0d1f80 0x7f3d3c0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d4184a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c161230 0x7f3d3c0d0130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d4184a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c0ce9a0 0x7f3d3c0d13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d4184a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c161640 con 0x7f3d3c161230 2024-04-25T19:11:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d4184a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c0d2de0 con 0x7f3d3c0d1f80 2024-04-25T19:11:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d4184a640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c06cf40 con 0x7f3d3c0ce9a0 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c161230 0x7f3d3c0d0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c161230 0x7f3d3c0d0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37486/0 (socket says 172.21.15.28:37486) 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 -- 172.21.15.28:0/1952735801 learned_addr learned my addr 172.21.15.28:0/1952735801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3affd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d3c0d1f80 0x7f3d3c0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3b7fe640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c0ce9a0 0x7f3d3c0d13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 -- 172.21.15.28:0/1952735801 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c0ce9a0 msgr2=0x7f3d3c0d13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c0ce9a0 0x7f3d3c0d13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 -- 172.21.15.28:0/1952735801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d3c0d1f80 msgr2=0x7f3d3c0ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d3c0d1f80 0x7f3d3c0ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 -- 172.21.15.28:0/1952735801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c0d0670 con 0x7f3d3c161230 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.917+0000 7f3d3a7fc640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c161230 0x7f3d3c0d0130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009890 tx=0x7f3d300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.918+0000 7f3d23fff640 1 -- 172.21.15.28:0/1952735801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30042070 con 0x7f3d3c161230 2024-04-25T19:11:57.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.918+0000 7f3d23fff640 1 -- 172.21.15.28:0/1952735801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3003e670 con 0x7f3d3c161230 2024-04-25T19:11:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.918+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c0d1900 con 0x7f3d3c161230 2024-04-25T19:11:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.918+0000 7f3d23fff640 1 -- 172.21.15.28:0/1952735801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30037040 con 0x7f3d3c161230 2024-04-25T19:11:57.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.918+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c0cf460 con 0x7f3d3c161230 2024-04-25T19:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.919+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3c050740 con 0x7f3d3c161230 2024-04-25T19:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.919+0000 7f3d23fff640 1 -- 172.21.15.28:0/1952735801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3d30045070 con 0x7f3d3c161230 2024-04-25T19:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.920+0000 7f3d23fff640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3d280744b0 0x7f3d280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.920+0000 7f3d3affd640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3d280744b0 0x7f3d280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.920+0000 7f3d23fff640 1 -- 172.21.15.28:0/1952735801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3d300b5d70 con 0x7f3d3c161230 2024-04-25T19:11:57.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.920+0000 7f3d3affd640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3d280744b0 0x7f3d280769c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c007fd0 tx=0x7f3d2c007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:57.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:57.925+0000 7f3d23fff640 1 -- 172.21.15.28:0/1952735801 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d3007fad0 con 0x7f3d3c161230 2024-04-25T19:11:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.117+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3d3c0cef10 con 0x7f3d3c161230 2024-04-25T19:11:58.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.117+0000 7f3d23fff640 1 -- 172.21.15.28:0/1952735801 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f3d30083dd0 con 0x7f3d3c161230 2024-04-25T19:11:58.120 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3d280744b0 msgr2=0x7f3d280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3d280744b0 0x7f3d280769c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c007fd0 tx=0x7f3d2c007de0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c161230 msgr2=0x7f3d3c0d0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c161230 0x7f3d3c0d0130 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009890 tx=0x7f3d300046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 shutdown_connections 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3d280744b0 0x7f3d280769c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3d3c0ce9a0 0x7f3d3c0d13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3c161230 0x7f3d3c0d0130 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 --2- 172.21.15.28:0/1952735801 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d3c0d1f80 0x7f3d3c0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.119+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 >> 172.21.15.28:0/1952735801 conn(0x7f3d3c073d30 msgr2=0x7f3d3c15abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.120+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 shutdown_connections 2024-04-25T19:11:58.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.120+0000 7f3d4184a640 1 -- 172.21.15.28:0/1952735801 wait complete. 2024-04-25T19:11:58.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:11:58.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.250+0000 7f9c65ef2640 1 Processor -- start 2024-04-25T19:11:58.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.251+0000 7f9c65ef2640 1 -- start start 2024-04-25T19:11:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.252+0000 7f9c65ef2640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c6009b540 0x7f9c6009b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.252+0000 7f9c65ef2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c6015e630 con 0x7f9c6015e220 2024-04-25T19:11:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.252+0000 7f9c65ef2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c6009be90 con 0x7f9c600cfa00 2024-04-25T19:11:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.252+0000 7f9c65ef2640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f9c600cffe0 con 0x7f9c6009b540 2024-04-25T19:11:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.252+0000 7f9c5effd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9c6015e220 0x7f9c60157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59054/0 (socket says 172.21.15.28:59054) 2024-04-25T19:11:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.252+0000 7f9c5effd640 1 -- 172.21.15.28:0/3488917264 learned_addr learned my addr 172.21.15.28:0/3488917264 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.252+0000 7f9c5ffff640 1 --2- 172.21.15.28:0/3488917264 >> v2:172.21.15.53:3300/0 conn(0x7f9c6009b540 0x7f9c6009b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.252+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349427629 0 0) 0x7f9c6015e630 con 0x7f9c6015e220 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c48003520 con 0x7f9c6015e220 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920891133 0 0) 0x7f9c50001110 con 0x7f9c600cfa00 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c6015e630 con 0x7f9c600cfa00 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3403833566 0 0) 0x7f9c48003520 con 0x7f9c6015e220 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 >> v2:172.21.15.53:3300/0 conn(0x7f9c6009b540 msgr2=0x7f9c6009b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 --2- 172.21.15.28:0/3488917264 >> v2:172.21.15.53:3300/0 conn(0x7f9c6009b540 0x7f9c6009b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 >> v1:172.21.15.53:6789/0 conn(0x7f9c600cfa00 legacy=0x7f9c600b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c600d02c0 con 0x7f9c6015e220 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f9c540015f0 con 0x7f9c6015e220 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 >> v1:172.21.15.28:6789/0 conn(0x7f9c6015e220 legacy=0x7f9c60157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c48003b90 0x7f9c48003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48004800 0x7f9c48004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48005280 0x7f9c4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c600cffe0 con 0x7f9c48005280 2024-04-25T19:11:58.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9c4800c230 con 0x7f9c48003b90 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9c4800c430 con 0x7f9c48004800 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5ffff640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48005280 0x7f9c4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5f7fe640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c48003b90 0x7f9c48003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5effd640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48004800 0x7f9c48004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5ffff640 1 -- 172.21.15.28:0/3488917264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48004800 msgr2=0x7f9c48004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5ffff640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48004800 0x7f9c48004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5ffff640 1 -- 172.21.15.28:0/3488917264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c48003b90 msgr2=0x7f9c48003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5ffff640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c48003b90 0x7f9c48003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.253+0000 7f9c5ffff640 1 -- 172.21.15.28:0/3488917264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c48003520 con 0x7f9c48005280 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.254+0000 7f9c5ffff640 1 -- 172.21.15.28:0/3488917264 reap_dead start 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.254+0000 7f9c5ffff640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48005280 0x7f9c4800b9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009f90 tx=0x7f9c48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5893675e88829e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.254+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58035020 con 0x7f9c48005280 2024-04-25T19:11:58.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.254+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c58003b90 con 0x7f9c48005280 2024-04-25T19:11:58.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.254+0000 7f9c5e7fc640 1 -- 172.21.15.28:0/3488917264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58009c10 con 0x7f9c48005280 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.255+0000 7f9c65ef2640 1 -- 172.21.15.28:0/3488917264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48005280 msgr2=0x7f9c4800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.255+0000 7f9c65ef2640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48005280 0x7f9c4800b9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009f90 tx=0x7f9c48005830 comp rx=0 tx=0).stop 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.255+0000 7f9c65ef2640 1 -- 172.21.15.28:0/3488917264 shutdown_connections 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.255+0000 7f9c65ef2640 1 --2- 172.21.15.28:0/3488917264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c48005280 0x7f9c4800b9c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.255+0000 7f9c65ef2640 1 -- 172.21.15.28:0/3488917264 >> 172.21.15.28:0/3488917264 conn(0x7f9c600b7620 msgr2=0x7f9c600b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.255+0000 7f9c65ef2640 1 -- 172.21.15.28:0/3488917264 shutdown_connections 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.255+0000 7f9c65ef2640 1 -- 172.21.15.28:0/3488917264 wait complete. 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.257+0000 7f9c65ef2640 1 Processor -- start 2024-04-25T19:11:58.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.257+0000 7f9c65ef2640 1 -- start start 2024-04-25T19:11:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c65ef2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48005280 0x7f9c6009d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c65ef2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c601583a0 0x7f9c601f7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c65ef2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c60146ed0 0x7f9c601472e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c65ef2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6015d800 con 0x7f9c60146ed0 2024-04-25T19:11:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c65ef2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9c6015d680 con 0x7f9c601583a0 2024-04-25T19:11:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c65ef2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9c6009b950 con 0x7f9c48005280 2024-04-25T19:11:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c60146ed0 0x7f9c601472e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c60146ed0 0x7f9c601472e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37498/0 (socket says 172.21.15.28:37498) 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 -- 172.21.15.28:0/1713819278 learned_addr learned my addr 172.21.15.28:0/1713819278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5effd640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c601583a0 0x7f9c601f7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5f7fe640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48005280 0x7f9c6009d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 -- 172.21.15.28:0/1713819278 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48005280 msgr2=0x7f9c6009d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48005280 0x7f9c6009d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 -- 172.21.15.28:0/1713819278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c601583a0 msgr2=0x7f9c601f7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c601583a0 0x7f9c601f7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 -- 172.21.15.28:0/1713819278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c6014d2a0 con 0x7f9c60146ed0 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c5ffff640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c60146ed0 0x7f9c601472e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9c5802aee0 tx=0x7f9c58009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.259+0000 7f9c47fff640 1 -- 172.21.15.28:0/1713819278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58035070 con 0x7f9c60146ed0 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.260+0000 7f9c47fff640 1 -- 172.21.15.28:0/1713819278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c58041070 con 0x7f9c60146ed0 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.260+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c601f8670 con 0x7f9c60146ed0 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.260+0000 7f9c47fff640 1 -- 172.21.15.28:0/1713819278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c58009c10 con 0x7f9c60146ed0 2024-04-25T19:11:58.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.260+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c6009c8c0 con 0x7f9c60146ed0 2024-04-25T19:11:58.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.261+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c601f83e0 con 0x7f9c60146ed0 2024-04-25T19:11:58.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.261+0000 7f9c47fff640 1 -- 172.21.15.28:0/1713819278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9c58005d10 con 0x7f9c60146ed0 2024-04-25T19:11:58.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.262+0000 7f9c47fff640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9c4c0744e0 0x7f9c4c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.262+0000 7f9c47fff640 1 -- 172.21.15.28:0/1713819278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9c580b6050 con 0x7f9c60146ed0 2024-04-25T19:11:58.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.262+0000 7f9c5f7fe640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9c4c0744e0 0x7f9c4c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.262+0000 7f9c5f7fe640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9c4c0744e0 0x7f9c4c0769f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9c50000c00 tx=0x7f9c50009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.267+0000 7f9c47fff640 1 -- 172.21.15.28:0/1713819278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9c6014d2a0 con 0x7f9c60146ed0 2024-04-25T19:11:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.450+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9c60050740 con 0x7f9c60146ed0 2024-04-25T19:11:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.451+0000 7f9c47fff640 1 -- 172.21.15.28:0/1713819278 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f9c5807fee0 con 0x7f9c60146ed0 2024-04-25T19:11:58.453 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:11:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9c4c0744e0 msgr2=0x7f9c4c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9c4c0744e0 0x7f9c4c0769f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9c50000c00 tx=0x7f9c50009040 comp rx=0 tx=0).stop 2024-04-25T19:11:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c60146ed0 msgr2=0x7f9c601472e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c60146ed0 0x7f9c601472e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9c5802aee0 tx=0x7f9c58009750 comp rx=0 tx=0).stop 2024-04-25T19:11:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 shutdown_connections 2024-04-25T19:11:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9c4c0744e0 0x7f9c4c0769f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c60146ed0 0x7f9c601472e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c601583a0 0x7f9c601f7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 --2- 172.21.15.28:0/1713819278 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9c48005280 0x7f9c6009d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 >> 172.21.15.28:0/1713819278 conn(0x7f9c600b7620 msgr2=0x7f9c600b1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 shutdown_connections 2024-04-25T19:11:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.453+0000 7f9c65ef2640 1 -- 172.21.15.28:0/1713819278 wait complete. 2024-04-25T19:11:58.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:11:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.575+0000 7ff95b58f640 1 Processor -- start 2024-04-25T19:11:58.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.576+0000 7ff95b58f640 1 -- start start 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff95b58f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff954158580 0x7ff9540b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff95b58f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff9540b48e0 0x7ff954161b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff95b58f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9540b4cf0 con 0x7ff95415f250 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff95b58f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7ff95406ee30 con 0x7ff9540b48e0 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff95b58f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff95415e770 con 0x7ff954158580 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff959b2c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff9540b48e0 0x7ff954161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff959b2c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff9540b48e0 0x7ff954161b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37508/0 (socket says 172.21.15.28:37508) 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff959b2c640 1 -- 172.21.15.28:0/1205773204 learned_addr learned my addr 172.21.15.28:0/1205773204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff95932b640 1 --1- 172.21.15.28:0/1205773204 >> v1:172.21.15.53:6789/0 conn(0x7ff95415f250 0x7ff95406e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40310/0 (socket says 172.21.15.28:40310) 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.577+0000 7ff958b2a640 1 --2- 172.21.15.28:0/1205773204 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff954158580 0x7ff9540b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.578+0000 7ff94bfff640 1 -- 172.21.15.28:0/1205773204 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2302541766 0 0) 0x7ff94c001110 con 0x7ff95415f250 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.578+0000 7ff959b2c640 1 -- 172.21.15.28:0/1205773204 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff954158580 msgr2=0x7ff9540b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:58.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.578+0000 7ff959b2c640 1 --2- 172.21.15.28:0/1205773204 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff954158580 0x7ff9540b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.578+0000 7ff959b2c640 1 -- 172.21.15.28:0/1205773204 >> v1:172.21.15.53:6789/0 conn(0x7ff95415f250 legacy=0x7ff95406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.578+0000 7ff959b2c640 1 -- 172.21.15.28:0/1205773204 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff95415ea50 con 0x7ff9540b48e0 2024-04-25T19:11:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.578+0000 7ff959b2c640 1 --2- 172.21.15.28:0/1205773204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9540b48e0 0x7ff954161b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff950009e50 tx=0x7ff95002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d0398b0ba2e1c2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.578+0000 7ff94bfff640 1 -- 172.21.15.28:0/1205773204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950036070 con 0x7ff9540b48e0 2024-04-25T19:11:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff94bfff640 1 -- 172.21.15.28:0/1205773204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff950031600 con 0x7ff9540b48e0 2024-04-25T19:11:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff94bfff640 1 -- 172.21.15.28:0/1205773204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9500319d0 con 0x7ff9540b48e0 2024-04-25T19:11:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff95b58f640 1 -- 172.21.15.28:0/1205773204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9540b48e0 msgr2=0x7ff954161b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff95b58f640 1 --2- 172.21.15.28:0/1205773204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9540b48e0 0x7ff954161b70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff950009e50 tx=0x7ff95002a680 comp rx=0 tx=0).stop 2024-04-25T19:11:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff95b58f640 1 -- 172.21.15.28:0/1205773204 shutdown_connections 2024-04-25T19:11:58.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff95b58f640 1 --2- 172.21.15.28:0/1205773204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff9540b48e0 0x7ff954161b70 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff95b58f640 1 --2- 172.21.15.28:0/1205773204 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff954158580 0x7ff9540b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff95b58f640 1 -- 172.21.15.28:0/1205773204 >> 172.21.15.28:0/1205773204 conn(0x7ff9540b75d0 msgr2=0x7ff9540b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:58.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff95b58f640 1 -- 172.21.15.28:0/1205773204 shutdown_connections 2024-04-25T19:11:58.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.579+0000 7ff95b58f640 1 -- 172.21.15.28:0/1205773204 wait complete. 2024-04-25T19:11:58.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.581+0000 7ff95b58f640 1 Processor -- start 2024-04-25T19:11:58.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.581+0000 7ff95b58f640 1 -- start start 2024-04-25T19:11:58.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.582+0000 7ff95b58f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9540b48e0 0x7ff9540af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.582+0000 7ff95b58f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff954158580 0x7ff954151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.582+0000 7ff95b58f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff95415f250 0x7ff954156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.582+0000 7ff95b58f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9540b3fd0 con 0x7ff95415f250 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.582+0000 7ff95b58f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff954158990 con 0x7ff954158580 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.582+0000 7ff95b58f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff9540b4150 con 0x7ff9540b48e0 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.582+0000 7ff959b2c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff95415f250 0x7ff954156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff959b2c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff95415f250 0x7ff954156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37512/0 (socket says 172.21.15.28:37512) 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff959b2c640 1 -- 172.21.15.28:0/3971108278 learned_addr learned my addr 172.21.15.28:0/3971108278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff95932b640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9540b48e0 0x7ff9540af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff958b2a640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff954158580 0x7ff954151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff959b2c640 1 -- 172.21.15.28:0/3971108278 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9540b48e0 msgr2=0x7ff9540af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff959b2c640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9540b48e0 0x7ff9540af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff959b2c640 1 -- 172.21.15.28:0/3971108278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff954158580 msgr2=0x7ff954151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff959b2c640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff954158580 0x7ff954151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff959b2c640 1 -- 172.21.15.28:0/3971108278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9540aff40 con 0x7ff95415f250 2024-04-25T19:11:58.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff959b2c640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff95415f250 0x7ff954156d80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff950009e50 tx=0x7ff950004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff949ffb640 1 -- 172.21.15.28:0/3971108278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950041070 con 0x7ff95415f250 2024-04-25T19:11:58.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff949ffb640 1 -- 172.21.15.28:0/3971108278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff950003990 con 0x7ff95415f250 2024-04-25T19:11:58.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9540b0220 con 0x7ff95415f250 2024-04-25T19:11:58.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.583+0000 7ff949ffb640 1 -- 172.21.15.28:0/3971108278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff950036040 con 0x7ff95415f250 2024-04-25T19:11:58.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.584+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9540b07e0 con 0x7ff95415f250 2024-04-25T19:11:58.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.584+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9540afcb0 con 0x7ff95415f250 2024-04-25T19:11:58.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.585+0000 7ff949ffb640 1 -- 172.21.15.28:0/3971108278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff950007820 con 0x7ff95415f250 2024-04-25T19:11:58.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.585+0000 7ff949ffb640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9380744b0 0x7ff9380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.586+0000 7ff949ffb640 1 -- 172.21.15.28:0/3971108278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff9500b60c0 con 0x7ff95415f250 2024-04-25T19:11:58.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.586+0000 7ff95932b640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9380744b0 0x7ff9380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.586+0000 7ff95932b640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9380744b0 0x7ff9380769c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff94c000f60 tx=0x7ff94c007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.590+0000 7ff949ffb640 1 -- 172.21.15.28:0/3971108278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff95004b020 con 0x7ff95415f250 2024-04-25T19:11:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.816+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff954050740 con 0x7ff95415f250 2024-04-25T19:11:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.817+0000 7ff949ffb640 1 -- 172.21.15.28:0/3971108278 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7ff950080470 con 0x7ff95415f250 2024-04-25T19:11:58.819 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:58.819 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:51.367677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:11:58.819 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9380744b0 msgr2=0x7ff9380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9380744b0 0x7ff9380769c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff94c000f60 tx=0x7ff94c007c00 comp rx=0 tx=0).stop 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff95415f250 msgr2=0x7ff954156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff95415f250 0x7ff954156d80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff950009e50 tx=0x7ff950004a40 comp rx=0 tx=0).stop 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 shutdown_connections 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff9380744b0 0x7ff9380769c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff95415f250 0x7ff954156d80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff954158580 0x7ff954151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 --2- 172.21.15.28:0/3971108278 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff9540b48e0 0x7ff9540af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 >> 172.21.15.28:0/3971108278 conn(0x7ff9540b75d0 msgr2=0x7ff95415ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:58.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 shutdown_connections 2024-04-25T19:11:58.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.819+0000 7ff95b58f640 1 -- 172.21.15.28:0/3971108278 wait complete. 2024-04-25T19:11:58.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:11:58.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.943+0000 7f8261b96640 1 Processor -- start 2024-04-25T19:11:58.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.943+0000 7f8261b96640 1 -- start start 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.944+0000 7f8261b96640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f825c06edb0 0x7f825c0b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.944+0000 7f8261b96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c0d1b30 0x7f825c0d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.944+0000 7f8261b96640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f825c160c00 con 0x7f825c1616a0 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.944+0000 7f8261b96640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f825c09c190 con 0x7f825c06edb0 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.944+0000 7f8261b96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f825c09bc50 con 0x7f825c0d1b30 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825bfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c0d1b30 0x7f825c0d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825b7fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f825c06edb0 0x7f825c0b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825affd640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f825c1616a0 0x7f825c09b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:40322/0 (socket says 172.21.15.28:40322) 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825bfff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c0d1b30 0x7f825c0d1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:47366/0 (socket says 172.21.15.28:47366) 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825affd640 1 -- 172.21.15.28:0/1759783679 learned_addr learned my addr 172.21.15.28:0/1759783679 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825b7fe640 1 -- 172.21.15.28:0/1759783679 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c0d1b30 msgr2=0x7f825c0d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:58.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825b7fe640 1 --2- 172.21.15.28:0/1759783679 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c0d1b30 0x7f825c0d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825b7fe640 1 -- 172.21.15.28:0/1759783679 >> v1:172.21.15.53:6789/0 conn(0x7f825c1616a0 legacy=0x7f825c09b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825b7fe640 1 -- 172.21.15.28:0/1759783679 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c09beb0 con 0x7f825c06edb0 2024-04-25T19:11:58.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825a7fc640 1 -- 172.21.15.28:0/1759783679 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661671223 0 0) 0x7f825c160c00 con 0x7f825c1616a0 2024-04-25T19:11:58.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.945+0000 7f825bfff640 1 --2- 172.21.15.28:0/1759783679 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c0d1b30 0x7f825c0d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.946+0000 7f825b7fe640 1 --2- 172.21.15.28:0/1759783679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c06edb0 0x7f825c0b3c40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f824c0097d0 tx=0x7f824c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f2e386b8a3daffc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.946+0000 7f825a7fc640 1 -- 172.21.15.28:0/1759783679 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c037070 con 0x7f825c06edb0 2024-04-25T19:11:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.946+0000 7f825a7fc640 1 -- 172.21.15.28:0/1759783679 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824c032440 con 0x7f825c06edb0 2024-04-25T19:11:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.946+0000 7f825a7fc640 1 -- 172.21.15.28:0/1759783679 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c032810 con 0x7f825c06edb0 2024-04-25T19:11:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.947+0000 7f8261b96640 1 -- 172.21.15.28:0/1759783679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c06edb0 msgr2=0x7f825c0b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.947+0000 7f8261b96640 1 --2- 172.21.15.28:0/1759783679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c06edb0 0x7f825c0b3c40 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f824c0097d0 tx=0x7f824c02a1c0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.947+0000 7f8261b96640 1 -- 172.21.15.28:0/1759783679 shutdown_connections 2024-04-25T19:11:58.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.947+0000 7f8261b96640 1 --2- 172.21.15.28:0/1759783679 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c0d1b30 0x7f825c0d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.947+0000 7f8261b96640 1 --2- 172.21.15.28:0/1759783679 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c06edb0 0x7f825c0b3c40 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.947+0000 7f8261b96640 1 -- 172.21.15.28:0/1759783679 >> 172.21.15.28:0/1759783679 conn(0x7f825c0b7620 msgr2=0x7f825c0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.947+0000 7f8261b96640 1 -- 172.21.15.28:0/1759783679 shutdown_connections 2024-04-25T19:11:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.947+0000 7f8261b96640 1 -- 172.21.15.28:0/1759783679 wait complete. 2024-04-25T19:11:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.948+0000 7f8261b96640 1 Processor -- start 2024-04-25T19:11:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.949+0000 7f8261b96640 1 -- start start 2024-04-25T19:11:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f8261b96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825c06edb0 0x7f825c0cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f8261b96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c0d1b30 0x7f825c162da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f8261b96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c1616a0 0x7f825c1ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f8261b96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f825c0b4180 con 0x7f825c06edb0 2024-04-25T19:11:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f8261b96640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f825c161180 con 0x7f825c0d1b30 2024-04-25T19:11:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f8261b96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f825c0d2480 con 0x7f825c1616a0 2024-04-25T19:11:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825c06edb0 0x7f825c0cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825c06edb0 0x7f825c0cc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37526/0 (socket says 172.21.15.28:37526) 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825b7fe640 1 -- 172.21.15.28:0/2642959660 learned_addr learned my addr 172.21.15.28:0/2642959660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825affd640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c0d1b30 0x7f825c162da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825bfff640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c1616a0 0x7f825c1ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825b7fe640 1 -- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c1616a0 msgr2=0x7f825c1ed5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825b7fe640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c1616a0 0x7f825c1ed5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825b7fe640 1 -- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c0d1b30 msgr2=0x7f825c162da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825b7fe640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c0d1b30 0x7f825c162da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825b7fe640 1 -- 172.21.15.28:0/2642959660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c0ccc10 con 0x7f825c06edb0 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.950+0000 7f825bfff640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c1616a0 0x7f825c1ed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.951+0000 7f825b7fe640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825c06edb0 0x7f825c0cc3d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f824c000c00 tx=0x7f824c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.951+0000 7f8243fff640 1 -- 172.21.15.28:0/2642959660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c037040 con 0x7f825c06edb0 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.951+0000 7f8243fff640 1 -- 172.21.15.28:0/2642959660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824c004230 con 0x7f825c06edb0 2024-04-25T19:11:58.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.951+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c06cc00 con 0x7f825c06edb0 2024-04-25T19:11:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.951+0000 7f8243fff640 1 -- 172.21.15.28:0/2642959660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c003610 con 0x7f825c06edb0 2024-04-25T19:11:58.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.951+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c1edb00 con 0x7f825c06edb0 2024-04-25T19:11:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.952+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f825c050740 con 0x7f825c06edb0 2024-04-25T19:11:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.953+0000 7f8243fff640 1 -- 172.21.15.28:0/2642959660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f824c0037b0 con 0x7f825c06edb0 2024-04-25T19:11:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.953+0000 7f8243fff640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82480744b0 0x7f82480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.953+0000 7f8243fff640 1 -- 172.21.15.28:0/2642959660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f824c0b62e0 con 0x7f825c06edb0 2024-04-25T19:11:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.953+0000 7f825affd640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82480744b0 0x7f82480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:58.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.954+0000 7f825affd640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82480744b0 0x7f82480769c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f825c0ceb90 tx=0x7f8250007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:58.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:58.958+0000 7f8243fff640 1 -- 172.21.15.28:0/2642959660 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f824c04c070 con 0x7f825c06edb0 2024-04-25T19:11:59.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.146+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f825c0cece0 con 0x7f825c06edb0 2024-04-25T19:11:59.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.147+0000 7f8243fff640 1 -- 172.21.15.28:0/2642959660 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f824c080850 con 0x7f825c06edb0 2024-04-25T19:11:59.149 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:11:59.149 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:11:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82480744b0 msgr2=0x7f82480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82480744b0 0x7f82480769c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f825c0ceb90 tx=0x7f8250007de0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825c06edb0 msgr2=0x7f825c0cc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825c06edb0 0x7f825c0cc3d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f824c000c00 tx=0x7f824c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 shutdown_connections 2024-04-25T19:11:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82480744b0 0x7f82480769c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f825c1616a0 0x7f825c1ed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825c0d1b30 0x7f825c162da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 --2- 172.21.15.28:0/2642959660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825c06edb0 0x7f825c0cc3d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 >> 172.21.15.28:0/2642959660 conn(0x7f825c0b7620 msgr2=0x7f825c0703f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 shutdown_connections 2024-04-25T19:11:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.149+0000 7f8261b96640 1 -- 172.21.15.28:0/2642959660 wait complete. 2024-04-25T19:11:59.162 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:11:59.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:11:59.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.276+0000 7f4b7c5c0640 1 Processor -- start 2024-04-25T19:11:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.276+0000 7f4b7c5c0640 1 -- start start 2024-04-25T19:11:59.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.277+0000 7f4b7c5c0640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4b74160180 0x7f4b74163c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.277+0000 7f4b7c5c0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b740d2550 0x7f4b740d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.277+0000 7f4b7c5c0640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b74164bb0 con 0x7f4b740d06c0 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.277+0000 7f4b7c5c0640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f4b740d4510 con 0x7f4b740d2550 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.277+0000 7f4b7c5c0640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f4b740d2ea0 con 0x7f4b74160180 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.277+0000 7f4b7ab5d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b740d2550 0x7f4b740d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.277+0000 7f4b7ab5d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b740d2550 0x7f4b740d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37538/0 (socket says 172.21.15.28:37538) 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.277+0000 7f4b7ab5d640 1 -- 172.21.15.28:0/3258321285 learned_addr learned my addr 172.21.15.28:0/3258321285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.278+0000 7f4b7a35c640 1 --2- 172.21.15.28:0/3258321285 >> v2:172.21.15.53:3300/0 conn(0x7f4b74160180 0x7f4b74163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.278+0000 7f4b7ab5d640 1 -- 172.21.15.28:0/3258321285 >> v2:172.21.15.53:3300/0 conn(0x7f4b74160180 msgr2=0x7f4b74163c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.278+0000 7f4b7ab5d640 1 --2- 172.21.15.28:0/3258321285 >> v2:172.21.15.53:3300/0 conn(0x7f4b74160180 0x7f4b74163c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.278+0000 7f4b7ab5d640 1 -- 172.21.15.28:0/3258321285 >> v1:172.21.15.53:6789/0 conn(0x7f4b740d06c0 legacy=0x7f4b740b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.278+0000 7f4b7ab5d640 1 -- 172.21.15.28:0/3258321285 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b74164310 con 0x7f4b740d2550 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.278+0000 7f4b7ab5d640 1 --2- 172.21.15.28:0/3258321285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d2550 0x7f4b740d2960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4b740d4690 tx=0x7f4b6402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af351840d0cdbec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7935a640 1 -- 172.21.15.28:0/3258321285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64036070 con 0x7f4b740d2550 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7935a640 1 -- 172.21.15.28:0/3258321285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b64031600 con 0x7f4b740d2550 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7935a640 1 -- 172.21.15.28:0/3258321285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b640319d0 con 0x7f4b740d2550 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/3258321285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d2550 msgr2=0x7f4b740d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/3258321285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d2550 0x7f4b740d2960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4b740d4690 tx=0x7f4b6402a730 comp rx=0 tx=0).stop 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/3258321285 shutdown_connections 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/3258321285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d2550 0x7f4b740d2960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/3258321285 >> v2:172.21.15.53:3300/0 conn(0x7f4b74160180 0x7f4b74163c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/3258321285 >> 172.21.15.28:0/3258321285 conn(0x7f4b7406c750 msgr2=0x7f4b7406cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/3258321285 shutdown_connections 2024-04-25T19:11:59.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.279+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/3258321285 wait complete. 2024-04-25T19:11:59.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.281+0000 7f4b7c5c0640 1 Processor -- start 2024-04-25T19:11:59.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.281+0000 7f4b7c5c0640 1 -- start start 2024-04-25T19:11:59.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7c5c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d06c0 0x7f4b7415ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7c5c0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4b740d2550 0x7f4b740cb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7c5c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b74160180 0x7f4b740ba210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7c5c0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b740d0ad0 con 0x7f4b740d06c0 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7c5c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4b741648d0 con 0x7f4b74160180 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7c5c0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4b740d49a0 con 0x7f4b740d2550 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d06c0 0x7f4b7415ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d06c0 0x7f4b7415ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37550/0 (socket says 172.21.15.28:37550) 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 -- 172.21.15.28:0/2840083876 learned_addr learned my addr 172.21.15.28:0/2840083876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7ab5d640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b74160180 0x7f4b740ba210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b79b5b640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4b740d2550 0x7f4b740cb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 -- 172.21.15.28:0/2840083876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4b740d2550 msgr2=0x7f4b740cb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4b740d2550 0x7f4b740cb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 -- 172.21.15.28:0/2840083876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b74160180 msgr2=0x7f4b740ba210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b74160180 0x7f4b740ba210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 -- 172.21.15.28:0/2840083876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b740c8e20 con 0x7f4b740d06c0 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7a35c640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d06c0 0x7f4b7415ae70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00d5f0 tx=0x7f4b6c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b72ffd640 1 -- 172.21.15.28:0/2840083876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0125f0 con 0x7f4b740d06c0 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b72ffd640 1 -- 172.21.15.28:0/2840083876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b6c019070 con 0x7f4b740d06c0 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b740c9100 con 0x7f4b740d06c0 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.283+0000 7f4b72ffd640 1 -- 172.21.15.28:0/2840083876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c0129d0 con 0x7f4b740d06c0 2024-04-25T19:11:59.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.284+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b740c96c0 con 0x7f4b740d06c0 2024-04-25T19:11:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.285+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b74050740 con 0x7f4b740d06c0 2024-04-25T19:11:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.285+0000 7f4b72ffd640 1 -- 172.21.15.28:0/2840083876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4b6c012b70 con 0x7f4b740d06c0 2024-04-25T19:11:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.286+0000 7f4b72ffd640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4b600744e0 0x7f4b600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.286+0000 7f4b72ffd640 1 -- 172.21.15.28:0/2840083876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4b6c0957c0 con 0x7f4b740d06c0 2024-04-25T19:11:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.286+0000 7f4b79b5b640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4b600744e0 0x7f4b600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.286+0000 7f4b79b5b640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4b600744e0 0x7f4b600769f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4b740b7e20 tx=0x7f4b68009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.290+0000 7f4b72ffd640 1 -- 172.21.15.28:0/2840083876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b6c05f5d0 con 0x7f4b740d06c0 2024-04-25T19:11:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.474+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4b740d3f90 con 0x7f4b740d06c0 2024-04-25T19:11:59.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.475+0000 7f4b72ffd640 1 -- 172.21.15.28:0/2840083876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f4b6c0638d0 con 0x7f4b740d06c0 2024-04-25T19:11:59.477 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4b600744e0 msgr2=0x7f4b600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4b600744e0 0x7f4b600769f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4b740b7e20 tx=0x7f4b68009040 comp rx=0 tx=0).stop 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d06c0 msgr2=0x7f4b7415ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d06c0 0x7f4b7415ae70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c00d5f0 tx=0x7f4b6c00dab0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 shutdown_connections 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4b600744e0 0x7f4b600769f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b74160180 0x7f4b740ba210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4b740d2550 0x7f4b740cb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 --2- 172.21.15.28:0/2840083876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b740d06c0 0x7f4b7415ae70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 >> 172.21.15.28:0/2840083876 conn(0x7f4b7406c750 msgr2=0x7f4b740b65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 shutdown_connections 2024-04-25T19:11:59.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.477+0000 7f4b7c5c0640 1 -- 172.21.15.28:0/2840083876 wait complete. 2024-04-25T19:11:59.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:11:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.603+0000 7f7a49b1f640 1 Processor -- start 2024-04-25T19:11:59.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.603+0000 7f7a49b1f640 1 -- start start 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a49b1f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a440cfa00 0x7f7a4415e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a49b1f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7a4409b540 0x7f7a4409b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a49b1f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a4409be90 con 0x7f7a440b21d0 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a49b1f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f7a440cffe0 con 0x7f7a440cfa00 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a49b1f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f7a440d02c0 con 0x7f7a4409b540 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a437fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a440cfa00 0x7f7a4415e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a437fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a440cfa00 0x7f7a4415e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37558/0 (socket says 172.21.15.28:37558) 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a437fe640 1 -- 172.21.15.28:0/4274297838 learned_addr learned my addr 172.21.15.28:0/4274297838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.604+0000 7f7a42ffd640 1 --1- 172.21.15.28:0/4274297838 >> v1:172.21.15.28:6789/0 conn(0x7f7a440b21d0 0x7f7a44157570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59070/0 (socket says 172.21.15.28:59070) 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.605+0000 7f7a43fff640 1 --2- 172.21.15.28:0/4274297838 >> v2:172.21.15.53:3300/0 conn(0x7f7a4409b540 0x7f7a4409b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.605+0000 7f7a437fe640 1 -- 172.21.15.28:0/4274297838 >> v2:172.21.15.53:3300/0 conn(0x7f7a4409b540 msgr2=0x7f7a4409b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.605+0000 7f7a437fe640 1 --2- 172.21.15.28:0/4274297838 >> v2:172.21.15.53:3300/0 conn(0x7f7a4409b540 0x7f7a4409b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.605+0000 7f7a437fe640 1 -- 172.21.15.28:0/4274297838 >> v1:172.21.15.28:6789/0 conn(0x7f7a440b21d0 legacy=0x7f7a44157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.605+0000 7f7a427fc640 1 -- 172.21.15.28:0/4274297838 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 778439278 0 0) 0x7f7a38001110 con 0x7f7a440b21d0 2024-04-25T19:11:59.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.605+0000 7f7a437fe640 1 -- 172.21.15.28:0/4274297838 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4415da20 con 0x7f7a440cfa00 2024-04-25T19:11:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.605+0000 7f7a437fe640 1 --2- 172.21.15.28:0/4274297838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440cfa00 0x7f7a4415e220 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7a440cf4c0 tx=0x7f7a3402c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbdb49c2043617c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a427fc640 1 -- 172.21.15.28:0/4274297838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34033640 con 0x7f7a440cfa00 2024-04-25T19:11:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a427fc640 1 -- 172.21.15.28:0/4274297838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a34038070 con 0x7f7a440cfa00 2024-04-25T19:11:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a427fc640 1 -- 172.21.15.28:0/4274297838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34033a00 con 0x7f7a440cfa00 2024-04-25T19:11:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4274297838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440cfa00 msgr2=0x7f7a4415e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4274297838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440cfa00 0x7f7a4415e220 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7a440cf4c0 tx=0x7f7a3402c8b0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4274297838 shutdown_connections 2024-04-25T19:11:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4274297838 >> v2:172.21.15.53:3300/0 conn(0x7f7a4409b540 0x7f7a4409b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4274297838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440cfa00 0x7f7a4415e220 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4274297838 >> 172.21.15.28:0/4274297838 conn(0x7f7a440b7620 msgr2=0x7f7a440b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4274297838 shutdown_connections 2024-04-25T19:11:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.606+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4274297838 wait complete. 2024-04-25T19:11:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.608+0000 7f7a49b1f640 1 Processor -- start 2024-04-25T19:11:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.608+0000 7f7a49b1f640 1 -- start start 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.609+0000 7f7a49b1f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a4409b540 0x7f7a440d3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.609+0000 7f7a49b1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440b21d0 0x7f7a44160200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a49b1f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a440cfa00 0x7f7a44071570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a49b1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7a440b2760 con 0x7f7a440b21d0 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a49b1f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7a440b25e0 con 0x7f7a4409b540 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a49b1f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4406c570 con 0x7f7a440cfa00 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440b21d0 0x7f7a44160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440b21d0 0x7f7a44160200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37560/0 (socket says 172.21.15.28:37560) 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 -- 172.21.15.28:0/4283545612 learned_addr learned my addr 172.21.15.28:0/4283545612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a437fe640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a4409b540 0x7f7a440d3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a43fff640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a440cfa00 0x7f7a44071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 -- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a440cfa00 msgr2=0x7f7a44071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a440cfa00 0x7f7a44071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 -- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a4409b540 msgr2=0x7f7a440d3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a4409b540 0x7f7a440d3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 -- 172.21.15.28:0/4283545612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a440d3690 con 0x7f7a440b21d0 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a437fe640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a4409b540 0x7f7a440d3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a43fff640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a440cfa00 0x7f7a44071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a42ffd640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440b21d0 0x7f7a44160200 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c430 tx=0x7f7a3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a2bfff640 1 -- 172.21.15.28:0/4283545612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a380074e0 con 0x7f7a440b21d0 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a2bfff640 1 -- 172.21.15.28:0/4283545612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a380047e0 con 0x7f7a440b21d0 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a440d0d40 con 0x7f7a440b21d0 2024-04-25T19:11:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a2bfff640 1 -- 172.21.15.28:0/4283545612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38004bd0 con 0x7f7a440b21d0 2024-04-25T19:11:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.610+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a441f6080 con 0x7f7a440b21d0 2024-04-25T19:11:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.611+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a44050740 con 0x7f7a440b21d0 2024-04-25T19:11:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.612+0000 7f7a2bfff640 1 -- 172.21.15.28:0/4283545612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7a38011070 con 0x7f7a440b21d0 2024-04-25T19:11:59.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.613+0000 7f7a2bfff640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7a300744e0 0x7f7a300769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.613+0000 7f7a437fe640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7a300744e0 0x7f7a300769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.613+0000 7f7a2bfff640 1 -- 172.21.15.28:0/4283545612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7a38095160 con 0x7f7a440b21d0 2024-04-25T19:11:59.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.613+0000 7f7a437fe640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7a300744e0 0x7f7a300769f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7a34037000 tx=0x7f7a34032000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.618+0000 7f7a2bfff640 1 -- 172.21.15.28:0/4283545612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a4409be90 con 0x7f7a440b21d0 2024-04-25T19:11:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.802+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a440cf520 con 0x7f7a440b21d0 2024-04-25T19:11:59.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.803+0000 7f7a2bfff640 1 -- 172.21.15.28:0/4283545612 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f7a3805f5a0 con 0x7f7a440b21d0 2024-04-25T19:11:59.805 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:11:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7a300744e0 msgr2=0x7f7a300769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7a300744e0 0x7f7a300769f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7a34037000 tx=0x7f7a34032000 comp rx=0 tx=0).stop 2024-04-25T19:11:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440b21d0 msgr2=0x7f7a44160200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440b21d0 0x7f7a44160200 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800c430 tx=0x7f7a3800c8f0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 shutdown_connections 2024-04-25T19:11:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7a300744e0 0x7f7a300769f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7a440cfa00 0x7f7a44071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a440b21d0 0x7f7a44160200 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 --2- 172.21.15.28:0/4283545612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a4409b540 0x7f7a440d3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 >> 172.21.15.28:0/4283545612 conn(0x7f7a440b7620 msgr2=0x7f7a4406ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 shutdown_connections 2024-04-25T19:11:59.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.805+0000 7f7a49b1f640 1 -- 172.21.15.28:0/4283545612 wait complete. 2024-04-25T19:11:59.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:11:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.931+0000 7f09c98a8640 1 Processor -- start 2024-04-25T19:11:59.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.931+0000 7f09c98a8640 1 -- start start 2024-04-25T19:11:59.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.932+0000 7f09c98a8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09c40d2d10 0x7f09c40d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.932+0000 7f09c98a8640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f09c41686b0 0x7f09c416aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.932+0000 7f09c98a8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c40b92a0 con 0x7f09c40b3d00 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.932+0000 7f09c98a8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f09c406e690 con 0x7f09c40d2d10 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.932+0000 7f09c98a8640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f09c409cea0 con 0x7f09c41686b0 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.932+0000 7f09c2ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09c40d2d10 0x7f09c40d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.933+0000 7f09c2ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09c40d2d10 0x7f09c40d21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37566/0 (socket says 172.21.15.28:37566) 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.933+0000 7f09c2ffd640 1 -- 172.21.15.28:0/2485827202 learned_addr learned my addr 172.21.15.28:0/2485827202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.933+0000 7f09c37fe640 1 --2- 172.21.15.28:0/2485827202 >> v2:172.21.15.53:3300/0 conn(0x7f09c41686b0 0x7f09c416aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.933+0000 7f09c2ffd640 1 -- 172.21.15.28:0/2485827202 >> v2:172.21.15.53:3300/0 conn(0x7f09c41686b0 msgr2=0x7f09c416aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.933+0000 7f09c2ffd640 1 --2- 172.21.15.28:0/2485827202 >> v2:172.21.15.53:3300/0 conn(0x7f09c41686b0 0x7f09c416aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.933+0000 7f09c2ffd640 1 -- 172.21.15.28:0/2485827202 >> v1:172.21.15.28:6789/0 conn(0x7f09c40b3d00 legacy=0x7f09c40b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.933+0000 7f09c2ffd640 1 -- 172.21.15.28:0/2485827202 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c406e8e0 con 0x7f09c40d2d10 2024-04-25T19:11:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.933+0000 7f09c27fc640 1 -- 172.21.15.28:0/2485827202 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4131103330 0 0) 0x7f09c40b92a0 con 0x7f09c40b3d00 2024-04-25T19:11:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.934+0000 7f09c2ffd640 1 --2- 172.21.15.28:0/2485827202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40d21c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f09c40b8c00 tx=0x7f09b402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9604f8c5e8bfa0a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.934+0000 7f09c27fc640 1 -- 172.21.15.28:0/2485827202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4037070 con 0x7f09c40d2d10 2024-04-25T19:11:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.934+0000 7f09c27fc640 1 -- 172.21.15.28:0/2485827202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b4032440 con 0x7f09c40d2d10 2024-04-25T19:11:59.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.934+0000 7f09c27fc640 1 -- 172.21.15.28:0/2485827202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b4032810 con 0x7f09c40d2d10 2024-04-25T19:11:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.934+0000 7f09c98a8640 1 -- 172.21.15.28:0/2485827202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 msgr2=0x7f09c40d21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.934+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2485827202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40d21c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f09c40b8c00 tx=0x7f09b402a1e0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.935+0000 7f09c98a8640 1 -- 172.21.15.28:0/2485827202 shutdown_connections 2024-04-25T19:11:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.935+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2485827202 >> v2:172.21.15.53:3300/0 conn(0x7f09c41686b0 0x7f09c416aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.935+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2485827202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40d21c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.935+0000 7f09c98a8640 1 -- 172.21.15.28:0/2485827202 >> 172.21.15.28:0/2485827202 conn(0x7f09c40d0450 msgr2=0x7f09c40d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:11:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.935+0000 7f09c98a8640 1 -- 172.21.15.28:0/2485827202 shutdown_connections 2024-04-25T19:11:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.935+0000 7f09c98a8640 1 -- 172.21.15.28:0/2485827202 wait complete. 2024-04-25T19:11:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.936+0000 7f09c98a8640 1 Processor -- start 2024-04-25T19:11:59.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.936+0000 7f09c98a8640 1 -- start start 2024-04-25T19:11:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c98a8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09c40b3d00 0x7f09c415f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c98a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c98a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09c41686b0 0x7f09c40b7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c98a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09c40d3120 con 0x7f09c40d2d10 2024-04-25T19:11:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c98a8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f09c406ebc0 con 0x7f09c40b3d00 2024-04-25T19:11:59.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c98a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f09c40b4820 con 0x7f09c41686b0 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09bbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37568/0 (socket says 172.21.15.28:37568) 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09bbfff640 1 -- 172.21.15.28:0/2929579315 learned_addr learned my addr 172.21.15.28:0/2929579315 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c2ffd640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09c40b3d00 0x7f09c415f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c37fe640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09c41686b0 0x7f09c40b7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09bbfff640 1 -- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09c41686b0 msgr2=0x7f09c40b7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09bbfff640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09c41686b0 0x7f09c40b7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09bbfff640 1 -- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09c40b3d00 msgr2=0x7f09c415f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09bbfff640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09c40b3d00 0x7f09c415f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09bbfff640 1 -- 172.21.15.28:0/2929579315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c40b2510 con 0x7f09c40d2d10 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.938+0000 7f09c2ffd640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09c40b3d00 0x7f09c415f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:11:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.939+0000 7f09bbfff640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40b1870 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c410 tx=0x7f09ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.939+0000 7f09c0ff9640 1 -- 172.21.15.28:0/2929579315 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ac017070 con 0x7f09c40d2d10 2024-04-25T19:11:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.939+0000 7f09c0ff9640 1 -- 172.21.15.28:0/2929579315 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09ac0035f0 con 0x7f09c40d2d10 2024-04-25T19:11:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.939+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c4160b60 con 0x7f09c40d2d10 2024-04-25T19:11:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.939+0000 7f09c0ff9640 1 -- 172.21.15.28:0/2929579315 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ac0039e0 con 0x7f09c40d2d10 2024-04-25T19:11:59.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.939+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c4161120 con 0x7f09c40d2d10 2024-04-25T19:11:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.940+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c40699f0 con 0x7f09c40d2d10 2024-04-25T19:11:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.941+0000 7f09c0ff9640 1 -- 172.21.15.28:0/2929579315 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f09ac01a070 con 0x7f09c40d2d10 2024-04-25T19:11:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.941+0000 7f09c0ff9640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09a40744b0 0x7f09a40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:11:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.941+0000 7f09c2ffd640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09a40744b0 0x7f09a40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:11:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.941+0000 7f09c0ff9640 1 -- 172.21.15.28:0/2929579315 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f09ac094770 con 0x7f09c40d2d10 2024-04-25T19:11:59.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.942+0000 7f09c2ffd640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09a40744b0 0x7f09a40769c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f09b4034000 tx=0x7f09b402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:11:59.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:11:59.946+0000 7f09c0ff9640 1 -- 172.21.15.28:0/2929579315 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09c40699f0 con 0x7f09c40d2d10 2024-04-25T19:12:00.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.131+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f09c4050740 con 0x7f09c40d2d10 2024-04-25T19:12:00.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.131+0000 7f09c0ff9640 1 -- 172.21.15.28:0/2929579315 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f09ac05e750 con 0x7f09c40d2d10 2024-04-25T19:12:00.133 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:00.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09a40744b0 msgr2=0x7f09a40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09a40744b0 0x7f09a40769c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f09b4034000 tx=0x7f09b402e000 comp rx=0 tx=0).stop 2024-04-25T19:12:00.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 msgr2=0x7f09c40b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40b1870 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c410 tx=0x7f09ac00c8d0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 shutdown_connections 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09a40744b0 0x7f09a40769c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09c41686b0 0x7f09c40b7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c40d2d10 0x7f09c40b1870 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 --2- 172.21.15.28:0/2929579315 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09c40b3d00 0x7f09c415f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 >> 172.21.15.28:0/2929579315 conn(0x7f09c40d0450 msgr2=0x7f09c4162fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 shutdown_connections 2024-04-25T19:12:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.133+0000 7f09c98a8640 1 -- 172.21.15.28:0/2929579315 wait complete. 2024-04-25T19:12:00.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:00.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.259+0000 7f6ab7f84640 1 Processor -- start 2024-04-25T19:12:00.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.259+0000 7f6ab7f84640 1 -- start start 2024-04-25T19:12:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab7f84640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6ab006edb0 0x7f6ab00b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab7f84640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6ab01616a0 0x7f6ab009b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab7f84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab00d1b30 0x7f6ab00d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab7f84640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f6ab009c190 con 0x7f6ab01616a0 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab7f84640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f6ab0160c00 con 0x7f6ab006edb0 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab7f84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab009beb0 con 0x7f6ab00d1b30 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab551f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6ab01616a0 0x7f6ab009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab551f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6ab01616a0 0x7f6ab009b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37578/0 (socket says 172.21.15.28:37578) 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab551f640 1 -- 172.21.15.28:0/860193582 learned_addr learned my addr 172.21.15.28:0/860193582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.260+0000 7f6ab5d20640 1 --2- 172.21.15.28:0/860193582 >> v2:172.21.15.53:3300/0 conn(0x7f6ab006edb0 0x7f6ab00b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.261+0000 7f6ab6521640 1 --2- 172.21.15.28:0/860193582 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab00d1b30 0x7f6ab00d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.261+0000 7f6ab551f640 1 -- 172.21.15.28:0/860193582 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab00d1b30 msgr2=0x7f6ab00d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.261+0000 7f6ab551f640 1 --2- 172.21.15.28:0/860193582 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab00d1b30 0x7f6ab00d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.261+0000 7f6ab551f640 1 -- 172.21.15.28:0/860193582 >> v2:172.21.15.53:3300/0 conn(0x7f6ab006edb0 msgr2=0x7f6ab00b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.261+0000 7f6ab551f640 1 --2- 172.21.15.28:0/860193582 >> v2:172.21.15.53:3300/0 conn(0x7f6ab006edb0 0x7f6ab00b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.261+0000 7f6ab551f640 1 -- 172.21.15.28:0/860193582 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab0160ea0 con 0x7f6ab01616a0 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.261+0000 7f6ab551f640 1 --2- 172.21.15.28:0/860193582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab01616a0 0x7f6ab009b540 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009870 tx=0x7f6aa002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6761294f1a3b101f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab4d1e640 1 -- 172.21.15.28:0/860193582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0037070 con 0x7f6ab01616a0 2024-04-25T19:12:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab4d1e640 1 -- 172.21.15.28:0/860193582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa0032440 con 0x7f6ab01616a0 2024-04-25T19:12:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab4d1e640 1 -- 172.21.15.28:0/860193582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0032810 con 0x7f6ab01616a0 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 -- 172.21.15.28:0/860193582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab01616a0 msgr2=0x7f6ab009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/860193582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab01616a0 0x7f6ab009b540 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0009870 tx=0x7f6aa002a210 comp rx=0 tx=0).stop 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 -- 172.21.15.28:0/860193582 shutdown_connections 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/860193582 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab00d1b30 0x7f6ab00d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/860193582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab01616a0 0x7f6ab009b540 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/860193582 >> v2:172.21.15.53:3300/0 conn(0x7f6ab006edb0 0x7f6ab00b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 -- 172.21.15.28:0/860193582 >> 172.21.15.28:0/860193582 conn(0x7f6ab00b7620 msgr2=0x7f6ab00b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 -- 172.21.15.28:0/860193582 shutdown_connections 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.262+0000 7f6ab7f84640 1 -- 172.21.15.28:0/860193582 wait complete. 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.264+0000 7f6ab7f84640 1 Processor -- start 2024-04-25T19:12:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.264+0000 7f6ab7f84640 1 -- start start 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab7f84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab006edb0 0x7f6ab0162970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab7f84640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab00d1b30 0x7f6ab00b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab7f84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab01616a0 0x7f6ab00af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab7f84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab00d2480 con 0x7f6ab006edb0 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab7f84640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6ab00b4180 con 0x7f6ab00d1b30 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab7f84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6ab009ba80 con 0x7f6ab01616a0 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab006edb0 0x7f6ab0162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab006edb0 0x7f6ab0162970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37582/0 (socket says 172.21.15.28:37582) 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 -- 172.21.15.28:0/356480342 learned_addr learned my addr 172.21.15.28:0/356480342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab551f640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab00d1b30 0x7f6ab00b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab6521640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab01616a0 0x7f6ab00af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 -- 172.21.15.28:0/356480342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab01616a0 msgr2=0x7f6ab00af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab01616a0 0x7f6ab00af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 -- 172.21.15.28:0/356480342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab00d1b30 msgr2=0x7f6ab00b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab00d1b30 0x7f6ab00b4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 -- 172.21.15.28:0/356480342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ab0153480 con 0x7f6ab006edb0 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6ab5d20640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab006edb0 0x7f6ab0162970 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00e840 tx=0x7f6aac00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6aa67fc640 1 -- 172.21.15.28:0/356480342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac017070 con 0x7f6ab006edb0 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.266+0000 7f6aa67fc640 1 -- 172.21.15.28:0/356480342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aac00c730 con 0x7f6ab006edb0 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.267+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ab0162eb0 con 0x7f6ab006edb0 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.267+0000 7f6aa67fc640 1 -- 172.21.15.28:0/356480342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aac00cb00 con 0x7f6ab006edb0 2024-04-25T19:12:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.267+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ab00b12a0 con 0x7f6ab006edb0 2024-04-25T19:12:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.267+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ab0050740 con 0x7f6ab006edb0 2024-04-25T19:12:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.268+0000 7f6aa67fc640 1 -- 172.21.15.28:0/356480342 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6aac01a070 con 0x7f6ab006edb0 2024-04-25T19:12:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.269+0000 7f6aa67fc640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6a940744b0 0x7f6a940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.269+0000 7f6aa67fc640 1 -- 172.21.15.28:0/356480342 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6aac094f80 con 0x7f6ab006edb0 2024-04-25T19:12:00.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.269+0000 7f6ab551f640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6a940744b0 0x7f6a940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.269+0000 7f6ab551f640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6a940744b0 0x7f6a940769c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0034000 tx=0x7f6aa002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.273+0000 7f6aa67fc640 1 -- 172.21.15.28:0/356480342 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6aac05ed70 con 0x7f6ab006edb0 2024-04-25T19:12:00.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.461+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6ab01631f0 con 0x7f6ab006edb0 2024-04-25T19:12:00.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.461+0000 7f6aa67fc640 1 -- 172.21.15.28:0/356480342 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f6aac063070 con 0x7f6ab006edb0 2024-04-25T19:12:00.463 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.463+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6a940744b0 msgr2=0x7f6a940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.463+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6a940744b0 0x7f6a940769c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0034000 tx=0x7f6aa002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.463+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab006edb0 msgr2=0x7f6ab0162970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.463+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab006edb0 0x7f6ab0162970 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00e840 tx=0x7f6aac00ed00 comp rx=0 tx=0).stop 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.464+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 shutdown_connections 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.464+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6a940744b0 0x7f6a940769c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.464+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ab01616a0 0x7f6ab00af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.464+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab00d1b30 0x7f6ab00b4d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.464+0000 7f6ab7f84640 1 --2- 172.21.15.28:0/356480342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ab006edb0 0x7f6ab0162970 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.464+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 >> 172.21.15.28:0/356480342 conn(0x7f6ab00b7620 msgr2=0x7f6ab00b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.464+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 shutdown_connections 2024-04-25T19:12:00.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.464+0000 7f6ab7f84640 1 -- 172.21.15.28:0/356480342 wait complete. 2024-04-25T19:12:00.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:00.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.586+0000 7fbe401d5640 1 Processor -- start 2024-04-25T19:12:00.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.586+0000 7fbe401d5640 1 -- start start 2024-04-25T19:12:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.587+0000 7fbe401d5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380d4910 0x7fbe38161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.587+0000 7fbe401d5640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe38163e80 con 0x7fbe380b5130 2024-04-25T19:12:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.587+0000 7fbe401d5640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe38164160 con 0x7fbe3806e5e0 2024-04-25T19:12:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.587+0000 7fbe401d5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbe38163c30 con 0x7fbe380d4910 2024-04-25T19:12:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3df71640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbe380b5130 0x7fbe3806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59090/0 (socket says 172.21.15.28:59090) 2024-04-25T19:12:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3df71640 1 -- 172.21.15.28:0/2292300697 learned_addr learned my addr 172.21.15.28:0/2292300697 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3e772640 1 --2- 172.21.15.28:0/2292300697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380d4910 0x7fbe38161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3cf6f640 1 -- 172.21.15.28:0/2292300697 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 576503081 0 0) 0x7fbe38163e80 con 0x7fbe380b5130 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3cf6f640 1 -- 172.21.15.28:0/2292300697 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe24003520 con 0x7fbe380b5130 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3cf6f640 1 -- 172.21.15.28:0/2292300697 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2706254163 0 0) 0x7fbe38164160 con 0x7fbe3806e5e0 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3cf6f640 1 -- 172.21.15.28:0/2292300697 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe38163e80 con 0x7fbe3806e5e0 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3e772640 1 -- 172.21.15.28:0/2292300697 >> v1:172.21.15.53:6789/0 conn(0x7fbe3806e5e0 legacy=0x7fbe3815f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3cf6f640 1 -- 172.21.15.28:0/2292300697 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2765286807 0 0) 0x7fbe24003520 con 0x7fbe380b5130 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3e772640 1 -- 172.21.15.28:0/2292300697 >> v1:172.21.15.28:6789/0 conn(0x7fbe380b5130 legacy=0x7fbe3806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.588+0000 7fbe3e772640 1 -- 172.21.15.28:0/2292300697 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe38164440 con 0x7fbe380d4910 2024-04-25T19:12:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.589+0000 7fbe3e772640 1 --2- 172.21.15.28:0/2292300697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380d4910 0x7fbe38161130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009dc0 tx=0x7fbe3002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e91c3f92d6b61918 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.589+0000 7fbe3cf6f640 1 -- 172.21.15.28:0/2292300697 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe34001690 con 0x7fbe380d4910 2024-04-25T19:12:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.589+0000 7fbe3cf6f640 1 -- 172.21.15.28:0/2292300697 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe30007540 con 0x7fbe380d4910 2024-04-25T19:12:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.589+0000 7fbe3cf6f640 1 -- 172.21.15.28:0/2292300697 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe300078f0 con 0x7fbe380d4910 2024-04-25T19:12:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.590+0000 7fbe401d5640 1 -- 172.21.15.28:0/2292300697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380d4910 msgr2=0x7fbe38161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.590+0000 7fbe401d5640 1 --2- 172.21.15.28:0/2292300697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380d4910 0x7fbe38161130 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbe30009dc0 tx=0x7fbe3002a660 comp rx=0 tx=0).stop 2024-04-25T19:12:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.590+0000 7fbe401d5640 1 -- 172.21.15.28:0/2292300697 shutdown_connections 2024-04-25T19:12:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.590+0000 7fbe401d5640 1 --2- 172.21.15.28:0/2292300697 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380d4910 0x7fbe38161130 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.590+0000 7fbe401d5640 1 -- 172.21.15.28:0/2292300697 >> 172.21.15.28:0/2292300697 conn(0x7fbe380b7ad0 msgr2=0x7fbe380b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.590+0000 7fbe401d5640 1 -- 172.21.15.28:0/2292300697 shutdown_connections 2024-04-25T19:12:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.590+0000 7fbe401d5640 1 -- 172.21.15.28:0/2292300697 wait complete. 2024-04-25T19:12:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.592+0000 7fbe401d5640 1 Processor -- start 2024-04-25T19:12:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.592+0000 7fbe401d5640 1 -- start start 2024-04-25T19:12:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe401d5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380b5130 0x7fbe38157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe401d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe380d4910 0x7fbe38162180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe401d5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe3815a740 0x7fbe381626c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe401d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3815fee0 con 0x7fbe380d4910 2024-04-25T19:12:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe401d5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38164a90 con 0x7fbe3815a740 2024-04-25T19:12:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe401d5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbe38160060 con 0x7fbe380b5130 2024-04-25T19:12:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3d770640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe380d4910 0x7fbe38162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3d770640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe380d4910 0x7fbe38162180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37588/0 (socket says 172.21.15.28:37588) 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3d770640 1 -- 172.21.15.28:0/992301850 learned_addr learned my addr 172.21.15.28:0/992301850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3e772640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe3815a740 0x7fbe381626c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3df71640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380b5130 0x7fbe38157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3d770640 1 -- 172.21.15.28:0/992301850 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380b5130 msgr2=0x7fbe38157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3d770640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380b5130 0x7fbe38157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3d770640 1 -- 172.21.15.28:0/992301850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe3815a740 msgr2=0x7fbe381626c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.593+0000 7fbe3d770640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe3815a740 0x7fbe381626c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.594+0000 7fbe3d770640 1 -- 172.21.15.28:0/992301850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe38159550 con 0x7fbe380d4910 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.594+0000 7fbe3d770640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe380d4910 0x7fbe38162180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800c430 tx=0x7fbe2800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.594+0000 7fbe2e7fc640 1 -- 172.21.15.28:0/992301850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28017070 con 0x7fbe380d4910 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.594+0000 7fbe2e7fc640 1 -- 172.21.15.28:0/992301850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe280052f0 con 0x7fbe380d4910 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.594+0000 7fbe2e7fc640 1 -- 172.21.15.28:0/992301850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe28003830 con 0x7fbe380d4910 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.594+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe38159830 con 0x7fbe380d4910 2024-04-25T19:12:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.594+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe38159df0 con 0x7fbe380d4910 2024-04-25T19:12:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.595+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe38050740 con 0x7fbe380d4910 2024-04-25T19:12:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.596+0000 7fbe2e7fc640 1 -- 172.21.15.28:0/992301850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbe2801a070 con 0x7fbe380d4910 2024-04-25T19:12:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.596+0000 7fbe2e7fc640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe1c0744e0 0x7fbe1c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.596+0000 7fbe3df71640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe1c0744e0 0x7fbe1c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.596+0000 7fbe2e7fc640 1 -- 172.21.15.28:0/992301850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbe28094900 con 0x7fbe380d4910 2024-04-25T19:12:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.597+0000 7fbe3df71640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe1c0744e0 0x7fbe1c0769f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbe34001800 tx=0x7fbe34001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.601+0000 7fbe2e7fc640 1 -- 172.21.15.28:0/992301850 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe2805e700 con 0x7fbe380d4910 2024-04-25T19:12:00.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.824+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe380d1b70 con 0x7fbe380d4910 2024-04-25T19:12:00.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.825+0000 7fbe2e7fc640 1 -- 172.21.15.28:0/992301850 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7fbe28062a00 con 0x7fbe380d4910 2024-04-25T19:12:00.827 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:00.827 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:51.367677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:12:00.827 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T19:12:00.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe1c0744e0 msgr2=0x7fbe1c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe1c0744e0 0x7fbe1c0769f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbe34001800 tx=0x7fbe34001630 comp rx=0 tx=0).stop 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe380d4910 msgr2=0x7fbe38162180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe380d4910 0x7fbe38162180 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbe2800c430 tx=0x7fbe2800c8f0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 shutdown_connections 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe1c0744e0 0x7fbe1c0769f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe3815a740 0x7fbe381626c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe380d4910 0x7fbe38162180 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 --2- 172.21.15.28:0/992301850 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe380b5130 0x7fbe38157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 >> 172.21.15.28:0/992301850 conn(0x7fbe380b7ad0 msgr2=0x7fbe380d5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 shutdown_connections 2024-04-25T19:12:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.827+0000 7fbe401d5640 1 -- 172.21.15.28:0/992301850 wait complete. 2024-04-25T19:12:00.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.952+0000 7f77b0814640 1 Processor -- start 2024-04-25T19:12:00.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.952+0000 7f77b0814640 1 -- start start 2024-04-25T19:12:00.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77b0814640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac0b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77b0814640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77ac164d60 con 0x7f77ac0b7650 2024-04-25T19:12:00.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77b0814640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77ac0d0220 con 0x7f77ac0b4e30 2024-04-25T19:12:00.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77b0814640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f77ac0d0b50 con 0x7f77ac0b9470 2024-04-25T19:12:00.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77aad9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f77ac0b7650 0x7f77ac0b7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37984/0 (socket says 172.21.15.28:37984) 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77aad9d640 1 -- 172.21.15.28:0/268551006 learned_addr learned my addr 172.21.15.28:0/268551006 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77a9d9b640 1 --2- 172.21.15.28:0/268551006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac0b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77a959a640 1 -- 172.21.15.28:0/268551006 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1925654524 0 0) 0x7f77ac164d60 con 0x7f77ac0b7650 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77a959a640 1 -- 172.21.15.28:0/268551006 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7794003520 con 0x7f77ac0b7650 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77a959a640 1 -- 172.21.15.28:0/268551006 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383889141 0 0) 0x7f77ac0d0220 con 0x7f77ac0b4e30 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77a959a640 1 -- 172.21.15.28:0/268551006 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f77ac164d60 con 0x7f77ac0b4e30 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77a9d9b640 1 -- 172.21.15.28:0/268551006 >> v1:172.21.15.53:6789/0 conn(0x7f77ac0b4e30 legacy=0x7f77ac0bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77a9d9b640 1 -- 172.21.15.28:0/268551006 >> v1:172.21.15.28:6789/0 conn(0x7f77ac0b7650 legacy=0x7f77ac0b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.954+0000 7f77a9d9b640 1 -- 172.21.15.28:0/268551006 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ac0d0500 con 0x7f77ac0b9470 2024-04-25T19:12:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.955+0000 7f77a9d9b640 1 --2- 172.21.15.28:0/268551006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac0b9880 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0d0cd0 tx=0x7f77a002a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=713827c176953ed0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.955+0000 7f77a959a640 1 -- 172.21.15.28:0/268551006 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a0037070 con 0x7f77ac0b9470 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.955+0000 7f77a959a640 1 -- 172.21.15.28:0/268551006 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a0032440 con 0x7f77ac0b9470 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.955+0000 7f77a959a640 1 -- 172.21.15.28:0/268551006 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00327f0 con 0x7f77ac0b9470 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.956+0000 7f77b0814640 1 -- 172.21.15.28:0/268551006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 msgr2=0x7f77ac0b9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.956+0000 7f77b0814640 1 --2- 172.21.15.28:0/268551006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac0b9880 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0d0cd0 tx=0x7f77a002a160 comp rx=0 tx=0).stop 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.956+0000 7f77b0814640 1 -- 172.21.15.28:0/268551006 shutdown_connections 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.956+0000 7f77b0814640 1 --2- 172.21.15.28:0/268551006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac0b9880 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.956+0000 7f77b0814640 1 -- 172.21.15.28:0/268551006 >> 172.21.15.28:0/268551006 conn(0x7f77ac0d2650 msgr2=0x7f77ac0d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.956+0000 7f77b0814640 1 -- 172.21.15.28:0/268551006 shutdown_connections 2024-04-25T19:12:00.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.956+0000 7f77b0814640 1 -- 172.21.15.28:0/268551006 wait complete. 2024-04-25T19:12:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.958+0000 7f77b0814640 1 Processor -- start 2024-04-25T19:12:00.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.958+0000 7f77b0814640 1 -- start start 2024-04-25T19:12:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77b0814640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ac0b4e30 0x7f77ac0cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77b0814640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ac0b7650 0x7f77ac1ea650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77b0814640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac162c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77b0814640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77ac1649c0 con 0x7f77ac0b7650 2024-04-25T19:12:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77b0814640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f77ac08e3a0 con 0x7f77ac0b4e30 2024-04-25T19:12:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77b0814640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f77ac164b40 con 0x7f77ac0b9470 2024-04-25T19:12:00.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ac0b7650 0x7f77ac1ea650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77a9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ac0b7650 0x7f77ac1ea650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44852/0 (socket says 172.21.15.28:44852) 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.959+0000 7f77a9d9b640 1 -- 172.21.15.28:0/4038983750 learned_addr learned my addr 172.21.15.28:0/4038983750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77aad9d640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac162c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77aa59c640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ac0b4e30 0x7f77ac0cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77a9d9b640 1 -- 172.21.15.28:0/4038983750 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 msgr2=0x7f77ac162c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77a9d9b640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac162c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77a9d9b640 1 -- 172.21.15.28:0/4038983750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ac0b4e30 msgr2=0x7f77ac0cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77a9d9b640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ac0b4e30 0x7f77ac0cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77a9d9b640 1 -- 172.21.15.28:0/4038983750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ac1f6850 con 0x7f77ac0b7650 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77a9d9b640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ac0b7650 0x7f77ac1ea650 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f77a00098c0 tx=0x7f77a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f779affd640 1 -- 172.21.15.28:0/4038983750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a0041020 con 0x7f77ac0b7650 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f779affd640 1 -- 172.21.15.28:0/4038983750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a003d620 con 0x7f77ac0b7650 2024-04-25T19:12:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f779affd640 1 -- 172.21.15.28:0/4038983750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a0037040 con 0x7f77ac0b7650 2024-04-25T19:12:00.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77ac1f6b10 con 0x7f77ac0b7650 2024-04-25T19:12:00.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.960+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77ac1f70d0 con 0x7f77ac0b7650 2024-04-25T19:12:00.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.961+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77ac0b4030 con 0x7f77ac0b7650 2024-04-25T19:12:00.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.962+0000 7f779affd640 1 -- 172.21.15.28:0/4038983750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f77a0044020 con 0x7f77ac0b7650 2024-04-25T19:12:00.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.962+0000 7f779affd640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f778c0744e0 0x7f778c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:00.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.962+0000 7f779affd640 1 -- 172.21.15.28:0/4038983750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f77a00b5d30 con 0x7f77ac0b7650 2024-04-25T19:12:00.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.962+0000 7f77aa59c640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f778c0744e0 0x7f778c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:00.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.963+0000 7f77aa59c640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f778c0744e0 0x7f778c0769f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f77a40011e0 tx=0x7f77a4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:00.967+0000 7f779affd640 1 -- 172.21.15.28:0/4038983750 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77ac0b4030 con 0x7f77ac0b7650 2024-04-25T19:12:01.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.154+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77ac050740 con 0x7f77ac0b7650 2024-04-25T19:12:01.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.155+0000 7f779affd640 1 -- 172.21.15.28:0/4038983750 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18454 (secure 0 0 0) 0x7f77a004b030 con 0x7f77ac0b7650 2024-04-25T19:12:01.157 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:01.157 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:11:34.348614+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f778c0744e0 msgr2=0x7f778c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f778c0744e0 0x7f778c0769f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f77a40011e0 tx=0x7f77a4009040 comp rx=0 tx=0).stop 2024-04-25T19:12:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ac0b7650 msgr2=0x7f77ac1ea650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ac0b7650 0x7f77ac1ea650 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f77a00098c0 tx=0x7f77a00046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 shutdown_connections 2024-04-25T19:12:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f778c0744e0 0x7f778c0769f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77ac0b9470 0x7f77ac162c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ac0b7650 0x7f77ac1ea650 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 --2- 172.21.15.28:0/4038983750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ac0b4e30 0x7f77ac0cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 >> 172.21.15.28:0/4038983750 conn(0x7f77ac0d2650 msgr2=0x7f77ac0b5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 shutdown_connections 2024-04-25T19:12:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.157+0000 7f77b0814640 1 -- 172.21.15.28:0/4038983750 wait complete. 2024-04-25T19:12:01.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:01.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.280+0000 7f97f2b57640 1 Processor -- start 2024-04-25T19:12:01.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.280+0000 7f97f2b57640 1 -- start start 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97f2b57640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97ec09cb60 0x7f97ec0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97f2b57640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97ec06d710 con 0x7f97ec0cef50 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97f2b57640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97ec08ce70 con 0x7f97ec09d010 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97f2b57640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f97ec08cc20 con 0x7f97ec09cb60 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97ebfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97ec09cb60 0x7f97ec0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97f10f4640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f97ec0cef50 0x7f97ec0d1cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37988/0 (socket says 172.21.15.28:37988) 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97ebfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97ec09cb60 0x7f97ec0cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44858/0 (socket says 172.21.15.28:44858) 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97f10f4640 1 -- 172.21.15.28:0/3870825389 learned_addr learned my addr 172.21.15.28:0/3870825389 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97ebfff640 1 -- 172.21.15.28:0/3870825389 >> v1:172.21.15.53:6789/0 conn(0x7f97ec09d010 legacy=0x7f97ec06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97ebfff640 1 -- 172.21.15.28:0/3870825389 >> v1:172.21.15.28:6789/0 conn(0x7f97ec0cef50 legacy=0x7f97ec0d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.282+0000 7f97ebfff640 1 -- 172.21.15.28:0/3870825389 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec0b92d0 con 0x7f97ec09cb60 2024-04-25T19:12:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.283+0000 7f97ebfff640 1 --2- 172.21.15.28:0/3870825389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec0cea10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97ec08cda0 tx=0x7f97d402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe643906525eb52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.283+0000 7f97eb7fe640 1 -- 172.21.15.28:0/3870825389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4037070 con 0x7f97ec09cb60 2024-04-25T19:12:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.283+0000 7f97eb7fe640 1 -- 172.21.15.28:0/3870825389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97d4032440 con 0x7f97ec09cb60 2024-04-25T19:12:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.283+0000 7f97eb7fe640 1 -- 172.21.15.28:0/3870825389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4032810 con 0x7f97ec09cb60 2024-04-25T19:12:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.284+0000 7f97f2b57640 1 -- 172.21.15.28:0/3870825389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 msgr2=0x7f97ec0cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.284+0000 7f97f2b57640 1 --2- 172.21.15.28:0/3870825389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec0cea10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97ec08cda0 tx=0x7f97d402a210 comp rx=0 tx=0).stop 2024-04-25T19:12:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.284+0000 7f97f2b57640 1 -- 172.21.15.28:0/3870825389 shutdown_connections 2024-04-25T19:12:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.284+0000 7f97f2b57640 1 --2- 172.21.15.28:0/3870825389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec0cea10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.284+0000 7f97f2b57640 1 -- 172.21.15.28:0/3870825389 >> 172.21.15.28:0/3870825389 conn(0x7f97ec0d0430 msgr2=0x7f97ec0d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.284+0000 7f97f2b57640 1 -- 172.21.15.28:0/3870825389 shutdown_connections 2024-04-25T19:12:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.284+0000 7f97f2b57640 1 -- 172.21.15.28:0/3870825389 wait complete. 2024-04-25T19:12:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.286+0000 7f97f2b57640 1 Processor -- start 2024-04-25T19:12:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.286+0000 7f97f2b57640 1 -- start start 2024-04-25T19:12:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f2b57640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec1e01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f2b57640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97ec09d010 0x7f97ec1d4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f2b57640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97ec0cef50 0x7f97ec1e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f2b57640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec06ea20 con 0x7f97ec09cb60 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f2b57640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec06e5e0 con 0x7f97ec09d010 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f2b57640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f97ec06eba0 con 0x7f97ec0cef50 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f08f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec1e01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f08f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec1e01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44866/0 (socket says 172.21.15.28:44866) 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.287+0000 7f97f08f3640 1 -- 172.21.15.28:0/415835606 learned_addr learned my addr 172.21.15.28:0/415835606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97ebfff640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97ec09d010 0x7f97ec1d4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f10f4640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97ec0cef50 0x7f97ec1e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f08f3640 1 -- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97ec0cef50 msgr2=0x7f97ec1e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f08f3640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97ec0cef50 0x7f97ec1e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f08f3640 1 -- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97ec09d010 msgr2=0x7f97ec1d4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f08f3640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97ec09d010 0x7f97ec1d4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f08f3640 1 -- 172.21.15.28:0/415835606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec0c8c60 con 0x7f97ec09cb60 2024-04-25T19:12:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f10f4640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97ec0cef50 0x7f97ec1e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f08f3640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec1e01c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97e400c760 tx=0x7f97e400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97e97fa640 1 -- 172.21.15.28:0/415835606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4016020 con 0x7f97ec09cb60 2024-04-25T19:12:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97e97fa640 1 -- 172.21.15.28:0/415835606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e4006de0 con 0x7f97ec09cb60 2024-04-25T19:12:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ec1e5d10 con 0x7f97ec09cb60 2024-04-25T19:12:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97e97fa640 1 -- 172.21.15.28:0/415835606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4003c00 con 0x7f97ec09cb60 2024-04-25T19:12:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.288+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ec1e62d0 con 0x7f97ec09cb60 2024-04-25T19:12:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.289+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ec050740 con 0x7f97ec09cb60 2024-04-25T19:12:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.290+0000 7f97e97fa640 1 -- 172.21.15.28:0/415835606 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f97e400a7f0 con 0x7f97ec09cb60 2024-04-25T19:12:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.290+0000 7f97e97fa640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f97cc0744e0 0x7f97cc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.290+0000 7f97e97fa640 1 -- 172.21.15.28:0/415835606 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f97e4095150 con 0x7f97ec09cb60 2024-04-25T19:12:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.290+0000 7f97ebfff640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f97cc0744e0 0x7f97cc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.291+0000 7f97ebfff640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f97cc0744e0 0x7f97cc0769f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f97d4034000 tx=0x7f97d402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.295+0000 7f97e97fa640 1 -- 172.21.15.28:0/415835606 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97e4020020 con 0x7f97ec09cb60 2024-04-25T19:12:01.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.519+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97ec076100 con 0x7f97ec09cb60 2024-04-25T19:12:01.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.519+0000 7f97e97fa640 1 -- 172.21.15.28:0/415835606 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f97e405f640 con 0x7f97ec09cb60 2024-04-25T19:12:01.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:01.521 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:51.367677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:12:01.521 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T19:12:01.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f97cc0744e0 msgr2=0x7f97cc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f97cc0744e0 0x7f97cc0769f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f97d4034000 tx=0x7f97d402e000 comp rx=0 tx=0).stop 2024-04-25T19:12:01.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 msgr2=0x7f97ec1e01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:01.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec1e01c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97e400c760 tx=0x7f97e400cc20 comp rx=0 tx=0).stop 2024-04-25T19:12:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 shutdown_connections 2024-04-25T19:12:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f97cc0744e0 0x7f97cc0769f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f97ec0cef50 0x7f97ec1e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f97ec09d010 0x7f97ec1d4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 --2- 172.21.15.28:0/415835606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97ec09cb60 0x7f97ec1e01c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 >> 172.21.15.28:0/415835606 conn(0x7f97ec0d0430 msgr2=0x7f97ec06ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 shutdown_connections 2024-04-25T19:12:01.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:01.521+0000 7f97f2b57640 1 -- 172.21.15.28:0/415835606 wait complete. 2024-04-25T19:12:01.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T19:11:29.309189+0000', 'modified': '2024-04-25T19:11:51.367677+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4281}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4281': {'gid': 4281, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6835/3585928833', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 3585928833}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 3585928833}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4281, 'qdb_cluster': [4281]} 2024-04-25T19:12:01.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T19:12:01.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T19:12:02.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.054+0000 7f6d9c7d5640 1 Processor -- start 2024-04-25T19:12:02.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.055+0000 7f6d9c7d5640 1 -- start start 2024-04-25T19:12:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d9c7d5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d9415aaa0 0x7f6d940affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d9c7d5640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6d940d0ee0 0x7f6d940b4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d9c7d5640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d9406f5b0 con 0x7f6d940b4e50 2024-04-25T19:12:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d9c7d5640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f6d940af290 con 0x7f6d940d0ee0 2024-04-25T19:12:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d9c7d5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6d940d14a0 con 0x7f6d9415aaa0 2024-04-25T19:12:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d9ad72640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6d940b4e50 0x7f6d9415d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38002/0 (socket says 172.21.15.28:38002) 2024-04-25T19:12:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d9ad72640 1 -- 172.21.15.28:0/901080396 learned_addr learned my addr 172.21.15.28:0/901080396 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:02.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d9a571640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d9415aaa0 0x7f6d940affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.056+0000 7f6d99d70640 1 --2- 172.21.15.28:0/901080396 >> v2:172.21.15.53:3300/0 conn(0x7f6d940d0ee0 0x7f6d940b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1090519919 0 0) 0x7f6d9406f5b0 con 0x7f6d940b4e50 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d7c003520 con 0x7f6d940b4e50 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2088155324 0 0) 0x7f6d7c003520 con 0x7f6d940b4e50 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d9415aaa0 msgr2=0x7f6d940affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d9415aaa0 0x7f6d940affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 >> v2:172.21.15.53:3300/0 conn(0x7f6d940d0ee0 msgr2=0x7f6d940b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 --2- 172.21.15.28:0/901080396 >> v2:172.21.15.53:3300/0 conn(0x7f6d940d0ee0 0x7f6d940b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d940af4e0 con 0x7f6d940b4e50 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f6d840019f0 con 0x7f6d940b4e50 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 >> v1:172.21.15.28:6789/0 conn(0x7f6d940b4e50 legacy=0x7f6d9415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d9415aaa0 0x7f6d7c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d7c003f30 0x7f6d7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c004d40 0x7f6d7c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940af290 con 0x7f6d7c003f30 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940d14a0 con 0x7f6d7c004d40 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6d7c00bac0 con 0x7f6d9415aaa0 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d99d70640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d7c003f30 0x7f6d7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9a571640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d9415aaa0 0x7f6d7c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d9ad72640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c004d40 0x7f6d7c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d99d70640 1 -- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d9415aaa0 msgr2=0x7f6d7c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d99d70640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d9415aaa0 0x7f6d7c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d99d70640 1 -- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c004d40 msgr2=0x7f6d7c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d99d70640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c004d40 0x7f6d7c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.057+0000 7f6d99d70640 1 -- 172.21.15.28:0/901080396 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7c003520 con 0x7f6d7c003f30 2024-04-25T19:12:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.058+0000 7f6d99d70640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d7c003f30 0x7f6d7c004800 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009a90 tx=0x7f6d7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb4e0b611464fc01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.058+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88036070 con 0x7f6d7c003f30 2024-04-25T19:12:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.058+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d88031400 con 0x7f6d7c003f30 2024-04-25T19:12:02.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.058+0000 7f6d9956f640 1 -- 172.21.15.28:0/901080396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88009690 con 0x7f6d7c003f30 2024-04-25T19:12:02.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.059+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/901080396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d7c003f30 msgr2=0x7f6d7c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.059+0000 7f6d9c7d5640 1 --2- 172.21.15.28:0/901080396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d7c003f30 0x7f6d7c004800 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6d88009a90 tx=0x7f6d7c005830 comp rx=0 tx=0).stop 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.059+0000 7f6d9ad72640 1 -- 172.21.15.28:0/901080396 reap_dead start 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.059+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/901080396 shutdown_connections 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.059+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/901080396 >> 172.21.15.28:0/901080396 conn(0x7f6d940742c0 msgr2=0x7f6d9406c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.059+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/901080396 shutdown_connections 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.059+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/901080396 wait complete. 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.061+0000 7f6d9c7d5640 1 Processor -- start 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.061+0000 7f6d9c7d5640 1 -- start start 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d9c7d5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7c003f30 0x7f6d9415af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d9c7d5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94158730 0x7f6d9415b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d9c7d5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d941f66a0 0x7f6d941f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d9c7d5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940affc0 con 0x7f6d94158730 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d9c7d5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940afaa0 con 0x7f6d941f66a0 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d9c7d5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6d940b0140 con 0x7f6d7c003f30 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d99d70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94158730 0x7f6d9415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d99d70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94158730 0x7f6d9415b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44880/0 (socket says 172.21.15.28:44880) 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d99d70640 1 -- 172.21.15.28:0/2978892487 learned_addr learned my addr 172.21.15.28:0/2978892487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d9ad72640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d941f66a0 0x7f6d941f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.062+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7c003f30 0x7f6d9415af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d99d70640 1 -- 172.21.15.28:0/2978892487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7c003f30 msgr2=0x7f6d9415af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d99d70640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7c003f30 0x7f6d9415af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d99d70640 1 -- 172.21.15.28:0/2978892487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d941f66a0 msgr2=0x7f6d941f6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d99d70640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d941f66a0 0x7f6d941f6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d99d70640 1 -- 172.21.15.28:0/2978892487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d94143a00 con 0x7f6d94158730 2024-04-25T19:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d99d70640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94158730 0x7f6d9415b450 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6d88000c00 tx=0x7f6d88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d92ffd640 1 -- 172.21.15.28:0/2978892487 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88040020 con 0x7f6d94158730 2024-04-25T19:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d92ffd640 1 -- 172.21.15.28:0/2978892487 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d88003c60 con 0x7f6d94158730 2024-04-25T19:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/2978892487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d941f3090 con 0x7f6d94158730 2024-04-25T19:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d92ffd640 1 -- 172.21.15.28:0/2978892487 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88036040 con 0x7f6d94158730 2024-04-25T19:12:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.063+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/2978892487 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d941f3650 con 0x7f6d94158730 2024-04-25T19:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.065+0000 7f6d92ffd640 1 -- 172.21.15.28:0/2978892487 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6d88043020 con 0x7f6d94158730 2024-04-25T19:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.066+0000 7f6d92ffd640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d800744b0 0x7f6d800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.066+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d800744b0 0x7f6d800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.066+0000 7f6d92ffd640 1 -- 172.21.15.28:0/2978892487 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6d880b56c0 con 0x7f6d94158730 2024-04-25T19:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.066+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d800744b0 0x7f6d800769c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009b20 tx=0x7f6d8c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.067+0000 7f6da3322740 1 Processor -- start 2024-04-25T19:12:02.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.067+0000 7f6da3322740 1 -- start start 2024-04-25T19:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6da3322740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162deb6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6da3322740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162dee34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6da3322740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 0x56162debc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6da3322740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56162de20740 con 0x56162db1a270 2024-04-25T19:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6da3322740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56162ddbc3a0 con 0x56162daf13b0 2024-04-25T19:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6da3322740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x56162de1c700 con 0x56162daf9590 2024-04-25T19:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6d9a571640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162deb6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6d9a571640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162deb6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44882/0 (socket says 172.21.15.28:44882) 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.068+0000 7f6d9a571640 1 -- 172.21.15.28:0/3976059349 learned_addr learned my addr 172.21.15.28:0/3976059349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d9ad72640 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 0x56162debc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d99d70640 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162dee34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d9a571640 1 -- 172.21.15.28:0/3976059349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 msgr2=0x56162debc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d9a571640 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 0x56162debc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d9a571640 1 -- 172.21.15.28:0/3976059349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 msgr2=0x56162dee34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d9a571640 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162dee34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d9a571640 1 -- 172.21.15.28:0/3976059349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56162ddd8750 con 0x56162db1a270 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d9a571640 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162deb6690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c014f70 tx=0x7f6d8c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b57580c114dcc21c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d7bfff640 1 -- 172.21.15.28:0/3976059349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c004020 con 0x56162db1a270 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d7bfff640 1 -- 172.21.15.28:0/3976059349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d8c0041c0 con 0x56162db1a270 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.069+0000 7f6d7bfff640 1 -- 172.21.15.28:0/3976059349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c022630 con 0x56162db1a270 2024-04-25T19:12:02.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 -- 172.21.15.28:0/3976059349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 msgr2=0x56162deb6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162deb6690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c014f70 tx=0x7f6d8c017000 comp rx=0 tx=0).stop 2024-04-25T19:12:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 -- 172.21.15.28:0/3976059349 shutdown_connections 2024-04-25T19:12:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 0x56162debc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162dee34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 --2- 172.21.15.28:0/3976059349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162deb6690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 -- 172.21.15.28:0/3976059349 >> 172.21.15.28:0/3976059349 conn(0x56162db0e3c0 msgr2=0x56162debb9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 -- 172.21.15.28:0/3976059349 shutdown_connections 2024-04-25T19:12:02.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.070+0000 7f6da3322740 1 -- 172.21.15.28:0/3976059349 wait complete. 2024-04-25T19:12:02.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:12:02.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.071+0000 7f6da3322740 1 -- start start 2024-04-25T19:12:02.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.071+0000 7f6da3322740 1 Processor -- start 2024-04-25T19:12:02.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6da3322740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162defe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6da3322740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 0x56162dee5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6da3322740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162dee5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6da3322740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56162dda7fe0 con 0x56162db1a270 2024-04-25T19:12:02.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6da3322740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56162dee6d40 con 0x56162daf13b0 2024-04-25T19:12:02.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6da3322740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x56162dee85e0 con 0x56162daf9590 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6da3322740 10 client.0 _ll_register_callbacks cb 0x56162ddbe150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6d9ad72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162dee5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6d9ad72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162dee5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44888/0 (socket says 172.21.15.28:44888) 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.072+0000 7f6d9ad72640 1 -- 172.21.15.28:0/2495482160 learned_addr learned my addr 172.21.15.28:0/2495482160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d99d70640 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 0x56162dee5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162defe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d9ad72640 1 -- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 msgr2=0x56162dee5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d9ad72640 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 0x56162dee5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d9ad72640 1 -- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 msgr2=0x56162defe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d9ad72640 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162defe9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d9ad72640 1 -- 172.21.15.28:0/2495482160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56162db322d0 con 0x56162db1a270 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162defe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d9ad72640 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162dee5580 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6d84002230 tx=0x7f6d84007860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c435c9c9bdd1ca7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d797fa640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d797fa640 1 -- 172.21.15.28:0/2495482160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8400aaa0 con 0x56162db1a270 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d797fa640 1 -- 172.21.15.28:0/2495482160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6d8400ac40 con 0x56162db1a270 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6da3322740 1 -- 172.21.15.28:0/2495482160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d8c0041c0 con 0x56162db1a270 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d797fa640 1 -- 172.21.15.28:0/2495482160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f6d8400ae40 con 0x56162db1a270 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6da3322740 10 client.4812 fetch_fsmap learned FSMap version 9 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6da3322740 1 -- 172.21.15.28:0/2495482160 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x56162dbfc6b0 con 0x56162db1a270 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.073+0000 7f6d797fa640 1 -- 172.21.15.28:0/2495482160 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d8c0041c0 con 0x56162db1a270 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6d797fa640 1 -- 172.21.15.28:0/2495482160 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f6d84015440 con 0x56162db1a270 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6da3322740 10 client.4812 fetch_fsmap finished waiting for FSMap version 9 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6da3322740 10 client.4812 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6da3322740 20 client.4812 populate_metadata read hostname 'smithi028' 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6da3322740 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x56162dad8e40 0x56162dedf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6da3322740 4 client.4812 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:12:02.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6da3322740 1 -- 172.21.15.28:0/2495482160 --> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56162de2e350 con 0x56162dad8e40 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x56162dad8e40 0x56162dedf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6d797fa640 10 client.4812 ms_handle_connect on v2:172.21.15.53:6834/3585928833 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.074+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x56162dad8e40 0x56162dedf600 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f1a1e931b33152a server_cookie=116fd143c2689d32 in_seq=0 out_seq=0 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6d797fa640 1 -- 172.21.15.28:0/2495482160 <== mds.0 v2:172.21.15.53:6834/3585928833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56162de2e350 con 0x56162dad8e40 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6d797fa640 10 client.4812 handle_command_reply: tid=0 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 client.4812 shutdown 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 -- 172.21.15.28:0/2495482160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 msgr2=0x56162dee5580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162dee5580 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6d84002230 tx=0x7f6d84007860 comp rx=0 tx=0).stop 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 -- 172.21.15.28:0/2495482160 shutdown_connections 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x56162dad8e40 0x56162dedf600 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162db1a270 0x56162dee5580 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162daf9590 0x56162dee5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 --2- 172.21.15.28:0/2495482160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162defe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6d797fa640 0 client.4812 ms_handle_reset on v2:172.21.15.53:6834/3585928833 2024-04-25T19:12:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 -- 172.21.15.28:0/2495482160 >> 172.21.15.28:0/2495482160 conn(0x56162db0e3c0 msgr2=0x56162df04e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.075+0000 7f6da3322740 1 -- 172.21.15.28:0/2495482160 shutdown_connections 2024-04-25T19:12:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.076+0000 7f6da3322740 1 -- 172.21.15.28:0/2495482160 wait complete. 2024-04-25T19:12:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.076+0000 7f6da3322740 20 client.4812 trim_cache size 0 max 16384 2024-04-25T19:12:02.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:12:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.095+0000 7f6da3322740 1 Processor -- start 2024-04-25T19:12:02.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.095+0000 7f6da3322740 1 -- start start 2024-04-25T19:12:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6da3322740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf13b0 0x56162deea140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6da3322740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf9590 0x56162df02090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6da3322740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 0x56162df025d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6da3322740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d941ff350 con 0x56162daf13b0 2024-04-25T19:12:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6da3322740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6d941e74c0 con 0x56162daf9590 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6da3322740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6d940708f0 con 0x56162db1a270 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6d9a571640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf13b0 0x56162deea140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6d9a571640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf13b0 0x56162deea140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44890/0 (socket says 172.21.15.28:44890) 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6d9a571640 1 -- 172.21.15.28:0/2206639984 learned_addr learned my addr 172.21.15.28:0/2206639984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6d99d70640 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf9590 0x56162df02090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.096+0000 7f6d9ad72640 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 0x56162df025d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d9a571640 1 -- 172.21.15.28:0/2206639984 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 msgr2=0x56162df025d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 0x56162df025d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d9a571640 1 -- 172.21.15.28:0/2206639984 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf9590 msgr2=0x56162df02090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf9590 0x56162df02090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d9a571640 1 -- 172.21.15.28:0/2206639984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56162deb5590 con 0x56162daf13b0 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d9a571640 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf13b0 0x56162deea140 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c015c80 tx=0x7f6d8c017070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbe70c46cc3833d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d7b7fe640 1 -- 172.21.15.28:0/2206639984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c037630 con 0x56162daf13b0 2024-04-25T19:12:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d7b7fe640 1 -- 172.21.15.28:0/2206639984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d8c03c070 con 0x56162daf13b0 2024-04-25T19:12:02.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.097+0000 7f6d7b7fe640 1 -- 172.21.15.28:0/2206639984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0379d0 con 0x56162daf13b0 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 -- 172.21.15.28:0/2206639984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf13b0 msgr2=0x56162deea140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf13b0 0x56162deea140 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c015c80 tx=0x7f6d8c017070 comp rx=0 tx=0).stop 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 -- 172.21.15.28:0/2206639984 shutdown_connections 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 0x56162df025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf9590 0x56162df02090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 --2- 172.21.15.28:0/2206639984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf13b0 0x56162deea140 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 -- 172.21.15.28:0/2206639984 >> 172.21.15.28:0/2206639984 conn(0x56162db0e3c0 msgr2=0x56162defd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 -- 172.21.15.28:0/2206639984 shutdown_connections 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.098+0000 7f6da3322740 1 -- 172.21.15.28:0/2206639984 wait complete. 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.099+0000 7f6da3322740 1 -- start start 2024-04-25T19:12:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.099+0000 7f6da3322740 1 Processor -- start 2024-04-25T19:12:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6da3322740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162df04660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6da3322740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf9590 0x56162df86490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6da3322740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 0x56162df869d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6da3322740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56162dee7140 con 0x56162daf9590 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6da3322740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x56162df01cb0 con 0x56162daf13b0 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6da3322740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x56162dee7700 con 0x56162db1a270 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6da3322740 10 client.0 _ll_register_callbacks cb 0x7f6d84001930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d99d70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf9590 0x56162df86490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d9a571640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162df04660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d99d70640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf9590 0x56162df86490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44896/0 (socket says 172.21.15.28:44896) 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d99d70640 1 -- 172.21.15.28:0/3891126565 learned_addr learned my addr 172.21.15.28:0/3891126565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d9ad72640 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 0x56162df869d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d9a571640 1 -- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 msgr2=0x56162df869d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d9a571640 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 0x56162df869d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d9a571640 1 -- 172.21.15.28:0/3891126565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf9590 msgr2=0x56162df86490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d9a571640 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf9590 0x56162df86490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d9a571640 1 -- 172.21.15.28:0/3891126565 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56162df04ba0 con 0x56162daf13b0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.101+0000 7f6d99d70640 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf9590 0x56162df86490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6d9a571640 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162df04660 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009900 tx=0x7f6d8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8bac1a9973aade server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6d797fa640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6d797fa640 1 -- 172.21.15.28:0/3891126565 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c037840 con 0x56162daf13b0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6d797fa640 1 -- 172.21.15.28:0/3891126565 <== mon.1 v2:172.21.15.53:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6d8c0379e0 con 0x56162daf13b0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 1 -- 172.21.15.28:0/3891126565 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d8c03c070 con 0x56162daf13b0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6d797fa640 1 -- 172.21.15.28:0/3891126565 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7f6d8c0044f0 con 0x56162daf13b0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 10 client.4738 fetch_fsmap learned FSMap version 9 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 1 -- 172.21.15.28:0/3891126565 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x56162deba480 con 0x56162daf13b0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6d797fa640 1 -- 172.21.15.28:0/3891126565 <== mon.1 v2:172.21.15.53:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d8c03c070 con 0x56162daf13b0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6d797fa640 1 -- 172.21.15.28:0/3891126565 <== mon.1 v2:172.21.15.53:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7f6d8c005990 con 0x56162daf13b0 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 10 client.4738 fetch_fsmap finished waiting for FSMap version 9 2024-04-25T19:12:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 10 client.4738 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 20 client.4738 populate_metadata read hostname 'smithi028' 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x56162df82060 0x56162df82470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 4 client.4738 mds_command: new command op to 4281 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.102+0000 7f6da3322740 1 -- 172.21.15.28:0/3891126565 --> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] -- command(tid 0: {"prefix": "status"}) -- 0x56162de281c0 con 0x56162df82060 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.103+0000 7f6d99d70640 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x56162df82060 0x56162df82470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.103+0000 7f6d797fa640 10 client.4738 ms_handle_connect on v2:172.21.15.53:6834/3585928833 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.103+0000 7f6d99d70640 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x56162df82060 0x56162df82470 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=377acd386083f80a server_cookie=92c6e93e36dcc9ab in_seq=0 out_seq=0 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6d797fa640 1 -- 172.21.15.28:0/3891126565 <== mds.0 v2:172.21.15.53:6834/3585928833 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x56162de281c0 con 0x56162df82060 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6d797fa640 10 client.4738 handle_command_reply: tid=0 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 client.4738 shutdown 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 -- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 msgr2=0x56162df04660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162df04660 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009900 tx=0x7f6d8c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 -- 172.21.15.28:0/3891126565 shutdown_connections 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:6834/3585928833,v1:172.21.15.53:6835/3585928833] conn(0x56162df82060 0x56162df82470 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x56162db1a270 0x56162df869d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56162daf9590 0x56162df86490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 --2- 172.21.15.28:0/3891126565 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x56162daf13b0 0x56162df04660 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6d797fa640 0 client.4738 ms_handle_reset on v2:172.21.15.53:6834/3585928833 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 -- 172.21.15.28:0/3891126565 >> 172.21.15.28:0/3891126565 conn(0x56162db0e3c0 msgr2=0x56162dee5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 -- 172.21.15.28:0/3891126565 shutdown_connections 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 1 -- 172.21.15.28:0/3891126565 wait complete. 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6da3322740 20 client.4738 trim_cache size 0 max 16384 2024-04-25T19:12:02.110 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "5d8007d4-24a5-41be-aa4d-cd64fedac62a", 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4281, 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 32.189122564000002, 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 9, 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 24, 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 24, 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 39.632844599000002 2024-04-25T19:12:02.111 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:12:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d800744b0 msgr2=0x7f6d800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.104+0000 7f6d9c7d5640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d800744b0 0x7f6d800769c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009b20 tx=0x7f6d8c009190 comp rx=0 tx=0).stop 2024-04-25T19:12:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94158730 msgr2=0x7f6d9415b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94158730 0x7f6d9415b450 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6d88000c00 tx=0x7f6d88004730 comp rx=0 tx=0).stop 2024-04-25T19:12:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/2978892487 shutdown_connections 2024-04-25T19:12:02.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d800744b0 0x7f6d800769c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d941f66a0 0x7f6d941f6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d94158730 0x7f6d9415b450 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 --2- 172.21.15.28:0/2978892487 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7c003f30 0x7f6d9415af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/2978892487 >> 172.21.15.28:0/2978892487 conn(0x7f6d940742c0 msgr2=0x7f6d9415c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/2978892487 shutdown_connections 2024-04-25T19:12:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.105+0000 7f6d9c7d5640 1 -- 172.21.15.28:0/2978892487 wait complete. 2024-04-25T19:12:02.118 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T19:12:02.127 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T19:12:02.127 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T19:12:02.127 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T19:12:02.127 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T19:12:02.169 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T19:12:02.169 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-25T19:12:02.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.170 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:12:02.226 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T19:12:02.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T19:12:02.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-25T19:12:02.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.381 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:12:02.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-25T19:12:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-04-25T19:12:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:12:02.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.479 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-25T19:12:02.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-04-25T19:12:02.574 INFO:teuthology.orchestra.run.smithi053.stderr:Cannot find device "ceph-brx" 2024-04-25T19:12:02.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:12:02.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.576 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-25T19:12:02.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:02.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-04-25T19:12:02.706 INFO:teuthology.orchestra.run.smithi053.stderr:Cannot find device "ceph-brx" 2024-04-25T19:12:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:12:02.708 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-04-25T19:12:02.708 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T19:12:02.708 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:02.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T19:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.819+0000 7f0cae900640 1 Processor -- start 2024-04-25T19:12:02.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.819+0000 7f0cae900640 1 -- start start 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.820+0000 7f0cae900640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0ca80d2360 0x7f0ca809cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.820+0000 7f0cae900640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ca80b4f90 0x7f0ca809bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.820+0000 7f0cae900640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ca80cfba0 con 0x7f0ca80b4b80 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cae900640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca80cf000 con 0x7f0ca80b4f90 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cae900640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca80d2940 con 0x7f0ca80d2360 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cace9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ca80b4f90 0x7f0ca809bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cace9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ca80b4f90 0x7f0ca809bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44900/0 (socket says 172.21.15.28:44900) 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cace9d640 1 -- 172.21.15.28:0/1464106713 learned_addr learned my addr 172.21.15.28:0/1464106713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0ca7fff640 1 --2- 172.21.15.28:0/1464106713 >> v2:172.21.15.53:3300/0 conn(0x7f0ca80d2360 0x7f0ca809cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cace9d640 1 -- 172.21.15.28:0/1464106713 >> v2:172.21.15.53:3300/0 conn(0x7f0ca80d2360 msgr2=0x7f0ca809cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cace9d640 1 --2- 172.21.15.28:0/1464106713 >> v2:172.21.15.53:3300/0 conn(0x7f0ca80d2360 0x7f0ca809cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cace9d640 1 -- 172.21.15.28:0/1464106713 >> v1:172.21.15.53:6789/0 conn(0x7f0ca80b4b80 legacy=0x7f0ca80cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.821+0000 7f0cace9d640 1 -- 172.21.15.28:0/1464106713 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca80ce9b0 con 0x7f0ca80b4f90 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.822+0000 7f0cace9d640 1 --2- 172.21.15.28:0/1464106713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80b4f90 0x7f0ca809bae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8071430 tx=0x7f0ca002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88a60ed679cfbc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.822+0000 7f0ca6ffd640 1 -- 172.21.15.28:0/1464106713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0036070 con 0x7f0ca80b4f90 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.822+0000 7f0ca6ffd640 1 -- 172.21.15.28:0/1464106713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ca0031600 con 0x7f0ca80b4f90 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.822+0000 7f0ca6ffd640 1 -- 172.21.15.28:0/1464106713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00319d0 con 0x7f0ca80b4f90 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.823+0000 7f0cae900640 1 -- 172.21.15.28:0/1464106713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80b4f90 msgr2=0x7f0ca809bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.823+0000 7f0cae900640 1 --2- 172.21.15.28:0/1464106713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80b4f90 0x7f0ca809bae0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8071430 tx=0x7f0ca002a730 comp rx=0 tx=0).stop 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.823+0000 7f0cae900640 1 -- 172.21.15.28:0/1464106713 shutdown_connections 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.823+0000 7f0cae900640 1 --2- 172.21.15.28:0/1464106713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80b4f90 0x7f0ca809bae0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.823+0000 7f0cae900640 1 --2- 172.21.15.28:0/1464106713 >> v2:172.21.15.53:3300/0 conn(0x7f0ca80d2360 0x7f0ca809cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.823+0000 7f0cae900640 1 -- 172.21.15.28:0/1464106713 >> 172.21.15.28:0/1464106713 conn(0x7f0ca80b54b0 msgr2=0x7f0ca80b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:02.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.823+0000 7f0cae900640 1 -- 172.21.15.28:0/1464106713 shutdown_connections 2024-04-25T19:12:02.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.823+0000 7f0cae900640 1 -- 172.21.15.28:0/1464106713 wait complete. 2024-04-25T19:12:02.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.824+0000 7f0cae900640 1 Processor -- start 2024-04-25T19:12:02.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.825+0000 7f0cae900640 1 -- start start 2024-04-25T19:12:02.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cae900640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ca80b4b80 0x7f0ca8153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cae900640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ca80b4f90 0x7f0ca80ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cae900640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80d2360 0x7f0ca809f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cae900640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca80ce750 con 0x7f0ca80d2360 2024-04-25T19:12:02.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cae900640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca80ce5d0 con 0x7f0ca80b4b80 2024-04-25T19:12:02.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cae900640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca809c020 con 0x7f0ca80b4f90 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cace9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80d2360 0x7f0ca809f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cace9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80d2360 0x7f0ca809f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44912/0 (socket says 172.21.15.28:44912) 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cace9d640 1 -- 172.21.15.28:0/1678058453 learned_addr learned my addr 172.21.15.28:0/1678058453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0ca7fff640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ca80b4b80 0x7f0ca8153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0ca77fe640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ca80b4f90 0x7f0ca80ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cace9d640 1 -- 172.21.15.28:0/1678058453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ca80b4f90 msgr2=0x7f0ca80ca020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cace9d640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ca80b4f90 0x7f0ca80ca020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.826+0000 7f0cace9d640 1 -- 172.21.15.28:0/1678058453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ca80b4b80 msgr2=0x7f0ca8153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.827+0000 7f0cace9d640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ca80b4b80 0x7f0ca8153dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.827+0000 7f0cace9d640 1 -- 172.21.15.28:0/1678058453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca809e5c0 con 0x7f0ca80d2360 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.827+0000 7f0cace9d640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80d2360 0x7f0ca809f340 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0034040 tx=0x7f0ca0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.827+0000 7f0ca4ff9640 1 -- 172.21.15.28:0/1678058453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0036050 con 0x7f0ca80d2360 2024-04-25T19:12:02.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.827+0000 7f0ca4ff9640 1 -- 172.21.15.28:0/1678058453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ca0042070 con 0x7f0ca80d2360 2024-04-25T19:12:02.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.827+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca809e8a0 con 0x7f0ca80d2360 2024-04-25T19:12:02.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.827+0000 7f0ca4ff9640 1 -- 172.21.15.28:0/1678058453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0009510 con 0x7f0ca80d2360 2024-04-25T19:12:02.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.827+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca80c7900 con 0x7f0ca80d2360 2024-04-25T19:12:02.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.828+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ca8050740 con 0x7f0ca80d2360 2024-04-25T19:12:02.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.829+0000 7f0ca4ff9640 1 -- 172.21.15.28:0/1678058453 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0ca0007bf0 con 0x7f0ca80d2360 2024-04-25T19:12:02.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.829+0000 7f0ca4ff9640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c940744b0 0x7f0c940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:02.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.829+0000 7f0ca4ff9640 1 -- 172.21.15.28:0/1678058453 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0ca00b62c0 con 0x7f0ca80d2360 2024-04-25T19:12:02.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.829+0000 7f0ca7fff640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c940744b0 0x7f0c940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:02.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.830+0000 7f0ca7fff640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c940744b0 0x7f0c940769c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8161d00 tx=0x7f0c9c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:02.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:02.834+0000 7f0ca4ff9640 1 -- 172.21.15.28:0/1678058453 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ca0080720 con 0x7f0ca80d2360 2024-04-25T19:12:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.062+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0ca809c890 con 0x7f0ca80d2360 2024-04-25T19:12:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.063+0000 7f0ca4ff9640 1 -- 172.21.15.28:0/1678058453 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f0ca0080020 con 0x7f0ca80d2360 2024-04-25T19:12:03.065 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T19:12:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c940744b0 msgr2=0x7f0c940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c940744b0 0x7f0c940769c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8161d00 tx=0x7f0c9c008040 comp rx=0 tx=0).stop 2024-04-25T19:12:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80d2360 msgr2=0x7f0ca809f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80d2360 0x7f0ca809f340 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0034040 tx=0x7f0ca0004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 shutdown_connections 2024-04-25T19:12:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c940744b0 0x7f0c940769c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ca80d2360 0x7f0ca809f340 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0ca80b4f90 0x7f0ca80ca020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 --2- 172.21.15.28:0/1678058453 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ca80b4b80 0x7f0ca8153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 >> 172.21.15.28:0/1678058453 conn(0x7f0ca80b54b0 msgr2=0x7f0ca80714d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 shutdown_connections 2024-04-25T19:12:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03.065+0000 7f0cae900640 1 -- 172.21.15.28:0/1678058453 wait complete. 2024-04-25T19:12:03.078 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T19:12:03.078 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T19:12:03.079 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-25T19:12:03.079 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T19:12:03.079 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:03.079 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T19:12:03.079 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T19:12:03.079 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T19:12:03.079 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T19:12:03.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:03.079 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-25T19:12:03.095 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42737sec preferred_lft 42737sec 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:03.096 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-25T19:12:03.097 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T19:12:03.097 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:03.097 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:03.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-25T19:12:03.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-25T19:12:03.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-25T19:12:03.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T19:12:03.097 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:03.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:03.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:03.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:03.233 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T19:12:03.256 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-25T19:12:03.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:03.258 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-25T19:12:03.315 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-25T19:12:03.315 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T19:12:03.315 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T19:12:03.315 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T19:12:03.315 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T19:12:03.315 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:03.315 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:03.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T19:12:03.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T19:12:03.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T19:12:03.316 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:03.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:03.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:03.441 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:12:03.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:03.457 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-25T19:12:03.511 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:03.511 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:03.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T19:12:03.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T19:12:03.512 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:03.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:03.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:03.628 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T19:12:03.629 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:03.629 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:03.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T19:12:03.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T19:12:03.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T19:12:03.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T19:12:03.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T19:12:03.629 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:03.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:03.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:03.742 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:03.742 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:03.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-25T19:12:03.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T19:12:03.742 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:03.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:03.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:03.821 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:03.821 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:03.836 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T19:12:03.836 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:03.836 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:03.890 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T19:12:03.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:03.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:03.984 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:03.984 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:04.094 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:04.094 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:04.094 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.53:3300,172.21.15.53:3301" 2024-04-25T19:12:04.094 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@5d8007d4-24a5-41be-aa4d-cd64fedac62a.cephfs=/ 2024-04-25T19:12:04.094 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.53:3300/172.21.15.53:3301" will pass to kernel 2024-04-25T19:12:04.095 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T19:12:04.095 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-04-25T19:12:04.095 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T19:12:04.095 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T19:12:04.095 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T19:12:04.096 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T19:12:04.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:04.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:04.121 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:04.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:04.218 INFO:teuthology.orchestra.run.smithi028.stdout:client4750 2024-04-25T19:12:04.218 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:04.219 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:04.235 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:04.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:04.331 INFO:teuthology.orchestra.run.smithi028.stdout:client4750 2024-04-25T19:12:04.332 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:04.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:04.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:04.388 INFO:teuthology.orchestra.run.smithi028.stdout:client4750 2024-04-25T19:12:04.389 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5d8007d4-24a5-41be-aa4d-cd64fedac62a.client4750/status') 2024-04-25T19:12:04.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:04.447 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.4750 (3)172.21.15.28:0/2228356548 2024-04-25T19:12:04.447 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-04-25T19:12:04.447 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-04-25T19:12:04.447 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-04-25T19:12:04.447 INFO:teuthology.orchestra.run.smithi028.stderr:70 bytes copied, 2.936e-05 s, 2.4 MB/s 2024-04-25T19:12:04.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:04.454 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-04-25T19:12:04.454 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T19:12:04.454 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T19:12:04.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T19:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.564+0000 7f825c6a8640 1 Processor -- start 2024-04-25T19:12:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.564+0000 7f825c6a8640 1 -- start start 2024-04-25T19:12:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.565+0000 7f825c6a8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f825406e5e0 0x7f8254161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.565+0000 7f825c6a8640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f825415f7d0 0x7f825415fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.565+0000 7f825c6a8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8254163e80 con 0x7f82540b5130 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.565+0000 7f825c6a8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8254163c30 con 0x7f825406e5e0 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.565+0000 7f825c6a8640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f8254164a90 con 0x7f825415f7d0 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.566+0000 7f8259c43640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f825406e5e0 0x7f8254161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.566+0000 7f8259c43640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f825406e5e0 0x7f8254161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44928/0 (socket says 172.21.15.28:44928) 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.566+0000 7f8259c43640 1 -- 172.21.15.28:0/3641096180 learned_addr learned my addr 172.21.15.28:0/3641096180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.566+0000 7f825ac45640 1 --2- 172.21.15.28:0/3641096180 >> v2:172.21.15.53:3300/0 conn(0x7f825415f7d0 0x7f825415fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.566+0000 7f8259c43640 1 -- 172.21.15.28:0/3641096180 >> v2:172.21.15.53:3300/0 conn(0x7f825415f7d0 msgr2=0x7f825415fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.566+0000 7f8259c43640 1 --2- 172.21.15.28:0/3641096180 >> v2:172.21.15.53:3300/0 conn(0x7f825415f7d0 0x7f825415fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.566+0000 7f8259c43640 1 -- 172.21.15.28:0/3641096180 >> v1:172.21.15.53:6789/0 conn(0x7f82540b5130 legacy=0x7f825406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.566+0000 7f8259c43640 1 -- 172.21.15.28:0/3641096180 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8254164160 con 0x7f825406e5e0 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.567+0000 7f8259c43640 1 --2- 172.21.15.28:0/3641096180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825406e5e0 0x7f8254161130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8254163b70 tx=0x7f824402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e481698c466ee2fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.567+0000 7f8259442640 1 -- 172.21.15.28:0/3641096180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8244037070 con 0x7f825406e5e0 2024-04-25T19:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.567+0000 7f8259442640 1 -- 172.21.15.28:0/3641096180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8244032440 con 0x7f825406e5e0 2024-04-25T19:12:04.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.567+0000 7f8259442640 1 -- 172.21.15.28:0/3641096180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8244032810 con 0x7f825406e5e0 2024-04-25T19:12:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.567+0000 7f825c6a8640 1 -- 172.21.15.28:0/3641096180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825406e5e0 msgr2=0x7f8254161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.567+0000 7f825c6a8640 1 --2- 172.21.15.28:0/3641096180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825406e5e0 0x7f8254161130 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8254163b70 tx=0x7f824402a230 comp rx=0 tx=0).stop 2024-04-25T19:12:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.568+0000 7f825c6a8640 1 -- 172.21.15.28:0/3641096180 shutdown_connections 2024-04-25T19:12:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.568+0000 7f825c6a8640 1 --2- 172.21.15.28:0/3641096180 >> v2:172.21.15.53:3300/0 conn(0x7f825415f7d0 0x7f825415fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.568+0000 7f825c6a8640 1 --2- 172.21.15.28:0/3641096180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825406e5e0 0x7f8254161130 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.568+0000 7f825c6a8640 1 -- 172.21.15.28:0/3641096180 >> 172.21.15.28:0/3641096180 conn(0x7f82540b7ad0 msgr2=0x7f82540b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.568+0000 7f825c6a8640 1 -- 172.21.15.28:0/3641096180 shutdown_connections 2024-04-25T19:12:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.568+0000 7f825c6a8640 1 -- 172.21.15.28:0/3641096180 wait complete. 2024-04-25T19:12:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.569+0000 7f825c6a8640 1 Processor -- start 2024-04-25T19:12:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.570+0000 7f825c6a8640 1 -- start start 2024-04-25T19:12:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825c6a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82540b5130 0x7f82541512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825c6a8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825415f7d0 0x7f8254157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825c6a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825415a3e0 0x7f825415a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825c6a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8254160120 con 0x7f825415a3e0 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825c6a8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8254164720 con 0x7f825415f7d0 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825c6a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f82541602a0 con 0x7f82540b5130 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825415a3e0 0x7f825415a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825415a3e0 0x7f825415a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44934/0 (socket says 172.21.15.28:44934) 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 -- 172.21.15.28:0/2677907066 learned_addr learned my addr 172.21.15.28:0/2677907066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f8259c43640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825415f7d0 0x7f8254157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825a444640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82540b5130 0x7f82541512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 -- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82540b5130 msgr2=0x7f82541512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82540b5130 0x7f82541512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 -- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825415f7d0 msgr2=0x7f8254157c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825415f7d0 0x7f8254157c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 -- 172.21.15.28:0/2677907066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825415ad30 con 0x7f825415a3e0 2024-04-25T19:12:04.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f8259c43640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825415f7d0 0x7f8254157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.571+0000 7f825ac45640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825415a3e0 0x7f825415a7f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f824c007e90 tx=0x7f824c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.572+0000 7f824affd640 1 -- 172.21.15.28:0/2677907066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c014420 con 0x7f825415a3e0 2024-04-25T19:12:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.572+0000 7f824affd640 1 -- 172.21.15.28:0/2677907066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824c019070 con 0x7f825415a3e0 2024-04-25T19:12:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.572+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8254154d50 con 0x7f825415a3e0 2024-04-25T19:12:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.572+0000 7f824affd640 1 -- 172.21.15.28:0/2677907066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c014800 con 0x7f825415a3e0 2024-04-25T19:12:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.572+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8254158c50 con 0x7f825415a3e0 2024-04-25T19:12:04.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.573+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8254158e90 con 0x7f825415a3e0 2024-04-25T19:12:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.573+0000 7f824affd640 1 -- 172.21.15.28:0/2677907066 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f824c0149a0 con 0x7f825415a3e0 2024-04-25T19:12:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.574+0000 7f824affd640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82380744b0 0x7f82380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.574+0000 7f824affd640 1 -- 172.21.15.28:0/2677907066 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f824c095d80 con 0x7f825415a3e0 2024-04-25T19:12:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.574+0000 7f825a444640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82380744b0 0x7f82380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.574+0000 7f825a444640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82380744b0 0x7f82380769c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f825406ec10 tx=0x7f8250008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:04.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.579+0000 7f824affd640 1 -- 172.21.15.28:0/2677907066 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f825415ad30 con 0x7f825415a3e0 2024-04-25T19:12:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.805+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f8254050740 con 0x7f825415a3e0 2024-04-25T19:12:04.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.805+0000 7f824affd640 1 -- 172.21.15.28:0/2677907066 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f824c05fb80 con 0x7f825415a3e0 2024-04-25T19:12:04.807 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T19:12:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82380744b0 msgr2=0x7f82380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:04.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82380744b0 0x7f82380769c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f825406ec10 tx=0x7f8250008040 comp rx=0 tx=0).stop 2024-04-25T19:12:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825415a3e0 msgr2=0x7f825415a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825415a3e0 0x7f825415a7f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f824c007e90 tx=0x7f824c00e4b0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 shutdown_connections 2024-04-25T19:12:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f82380744b0 0x7f82380769c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f825415a3e0 0x7f825415a7f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825415f7d0 0x7f8254157c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 --2- 172.21.15.28:0/2677907066 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f82540b5130 0x7f82541512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:04.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 >> 172.21.15.28:0/2677907066 conn(0x7f82540b7ad0 msgr2=0x7f82540684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:04.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 shutdown_connections 2024-04-25T19:12:04.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:04.807+0000 7f825c6a8640 1 -- 172.21.15.28:0/2677907066 wait complete. 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T19:12:04.821 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T19:12:04.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:04.822 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-25T19:12:04.837 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:04.837 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:04.837 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42735sec preferred_lft 42735sec 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:04.838 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether b6:8b:9a:39:76:98 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::d015:3fff:fe07:1fb/64 scope link tentative 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether b6:8b:9a:39:76:98 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::b48b:9aff:fe39:7698/64 scope link tentative 2024-04-25T19:12:04.839 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:04.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:04.840 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T19:12:04.900 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-25T19:12:04.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:04.902 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-25T19:12:04.959 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-25T19:12:04.959 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T19:12:04.959 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T19:12:04.959 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T19:12:04.959 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T19:12:04.959 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:04.959 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:04.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T19:12:04.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T19:12:04.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T19:12:04.960 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:05.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:05.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:05.085 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:12:05.101 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T19:12:05.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:05.102 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-25T19:12:05.156 INFO:teuthology.orchestra.run.smithi028.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-25T19:12:05.157 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:05.157 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:05.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T19:12:05.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-25T19:12:05.157 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:05.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:05.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:05.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T19:12:05.297 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:05.297 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T19:12:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether b2:a2:eb:1e:72:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T19:12:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T19:12:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::b0a2:ebff:fe1e:7287/64 scope link 2024-04-25T19:12:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:05.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:05.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T19:12:05.362 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether b2:a2:eb:1e:72:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::b0a2:ebff:fe1e:7287/64 scope link 2024-04-25T19:12:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:05.364 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-25T19:12:05.365 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:05.365 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:05.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-25T19:12:05.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T19:12:05.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-25T19:12:05.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-25T19:12:05.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-25T19:12:05.365 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:05.516 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:05.516 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:05.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.1 up 2024-04-25T19:12:05.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T19:12:05.516 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:05.593 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:05.593 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-25T19:12:05.608 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-25T19:12:05.608 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:05.608 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-25T19:12:05.663 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-04-25T19:12:05.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:05.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:05.757 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:05.758 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:05.779 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:05.779 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:05.779 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.53:3300,172.21.15.53:3301" 2024-04-25T19:12:05.779 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 2@5d8007d4-24a5-41be-aa4d-cd64fedac62a.cephfs=/ 2024-04-25T19:12:05.779 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.28:3300/172.21.15.53:3300/172.21.15.53:3301" will pass to kernel 2024-04-25T19:12:05.780 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-04-25T19:12:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-04-25T19:12:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T19:12:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T19:12:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T19:12:05.782 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T19:12:05.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:05.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-25T19:12:05.806 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:05.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:05.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:05.905 INFO:teuthology.orchestra.run.smithi028.stdout:client4556 2024-04-25T19:12:05.905 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:05.905 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:05.921 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:06.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:06.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:06.020 INFO:teuthology.orchestra.run.smithi028.stdout:client4556 2024-04-25T19:12:06.020 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:06.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:06.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:06.077 INFO:teuthology.orchestra.run.smithi028.stdout:client4556 2024-04-25T19:12:06.078 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5d8007d4-24a5-41be-aa4d-cd64fedac62a.client4556/status') 2024-04-25T19:12:06.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:06.138 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.4556 (3)172.21.15.28:0/2553397305 2024-04-25T19:12:06.138 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-04-25T19:12:06.138 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-04-25T19:12:06.138 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-04-25T19:12:06.138 INFO:teuthology.orchestra.run.smithi028.stderr:70 bytes copied, 2.3862e-05 s, 2.9 MB/s 2024-04-25T19:12:06.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:06.145 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-04-25T19:12:06.145 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T19:12:06.145 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T19:12:06.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T19:12:06.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.254+0000 7fb053aca640 1 Processor -- start 2024-04-25T19:12:06.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.255+0000 7fb053aca640 1 -- start start 2024-04-25T19:12:06.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb053aca640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c09caf0 0x7fb04c0d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:06.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb053aca640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb04c0d2670 0x7fb04c0b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:06.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb053aca640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb04c0d1ce0 con 0x7fb04c0b7710 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb053aca640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb04c0b8fa0 con 0x7fb04c0d2670 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb053aca640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb04c0cf8d0 con 0x7fb04c09caf0 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb051065640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb04c0d2670 0x7fb04c0b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb052067640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb04c0b7710 0x7fb04c162610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38012/0 (socket says 172.21.15.28:38012) 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb052067640 1 -- 172.21.15.28:0/153362388 learned_addr learned my addr 172.21.15.28:0/153362388 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.256+0000 7fb051866640 1 --2- 172.21.15.28:0/153362388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c09caf0 0x7fb04c0d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.257+0000 7fb051065640 1 -- 172.21.15.28:0/153362388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c09caf0 msgr2=0x7fb04c0d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.257+0000 7fb051065640 1 --2- 172.21.15.28:0/153362388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c09caf0 0x7fb04c0d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.257+0000 7fb050864640 1 -- 172.21.15.28:0/153362388 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2943006039 0 0) 0x7fb04c0d1ce0 con 0x7fb04c0b7710 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.257+0000 7fb051065640 1 -- 172.21.15.28:0/153362388 >> v1:172.21.15.28:6789/0 conn(0x7fb04c0b7710 legacy=0x7fb04c162610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.257+0000 7fb051065640 1 -- 172.21.15.28:0/153362388 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04c0cfbb0 con 0x7fb04c0d2670 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.257+0000 7fb051866640 1 --2- 172.21.15.28:0/153362388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c09caf0 0x7fb04c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.257+0000 7fb051065640 1 --2- 172.21.15.28:0/153362388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c0d2670 0x7fb04c0b71d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb0400097a0 tx=0x7fb04002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a3e391d4d4db9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb050864640 1 -- 172.21.15.28:0/153362388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040037070 con 0x7fb04c0d2670 2024-04-25T19:12:06.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb050864640 1 -- 172.21.15.28:0/153362388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04002ae50 con 0x7fb04c0d2670 2024-04-25T19:12:06.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb050864640 1 -- 172.21.15.28:0/153362388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb040032660 con 0x7fb04c0d2670 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb053aca640 1 -- 172.21.15.28:0/153362388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c0d2670 msgr2=0x7fb04c0b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb053aca640 1 --2- 172.21.15.28:0/153362388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c0d2670 0x7fb04c0b71d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb0400097a0 tx=0x7fb04002a100 comp rx=0 tx=0).stop 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb053aca640 1 -- 172.21.15.28:0/153362388 shutdown_connections 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb053aca640 1 --2- 172.21.15.28:0/153362388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c0d2670 0x7fb04c0b71d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb053aca640 1 --2- 172.21.15.28:0/153362388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c09caf0 0x7fb04c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb053aca640 1 -- 172.21.15.28:0/153362388 >> 172.21.15.28:0/153362388 conn(0x7fb04c0d03f0 msgr2=0x7fb04c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb053aca640 1 -- 172.21.15.28:0/153362388 shutdown_connections 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.258+0000 7fb053aca640 1 -- 172.21.15.28:0/153362388 wait complete. 2024-04-25T19:12:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.260+0000 7fb053aca640 1 Processor -- start 2024-04-25T19:12:06.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.260+0000 7fb053aca640 1 -- start start 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.261+0000 7fb053aca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c09caf0 0x7fb04c1dbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.261+0000 7fb053aca640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c0b7710 0x7fb04c06e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb053aca640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c0d2670 0x7fb04c1e7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb053aca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c0d4790 con 0x7fb04c09caf0 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb053aca640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb04c0b9430 con 0x7fb04c0d2670 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb053aca640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb04c0d2b10 con 0x7fb04c0b7710 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c09caf0 0x7fb04c1dbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c09caf0 0x7fb04c1dbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44950/0 (socket says 172.21.15.28:44950) 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 -- 172.21.15.28:0/88662044 learned_addr learned my addr 172.21.15.28:0/88662044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051065640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c0b7710 0x7fb04c06e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb052067640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c0d2670 0x7fb04c1e7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 -- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c0b7710 msgr2=0x7fb04c06e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c0b7710 0x7fb04c06e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 -- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c0d2670 msgr2=0x7fb04c1e7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c0d2670 0x7fb04c1e7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 -- 172.21.15.28:0/88662044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb04c0c8320 con 0x7fb04c09caf0 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051065640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c0b7710 0x7fb04c06e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb051866640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c09caf0 0x7fb04c1dbce0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb04400cde0 tx=0x7fb04400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb04a7fc640 1 -- 172.21.15.28:0/88662044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04400f680 con 0x7fb04c09caf0 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb04a7fc640 1 -- 172.21.15.28:0/88662044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb04400f820 con 0x7fb04c09caf0 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb04a7fc640 1 -- 172.21.15.28:0/88662044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb04400fc10 con 0x7fb04c09caf0 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb04c0c8600 con 0x7fb04c09caf0 2024-04-25T19:12:06.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.262+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb04c0c8bc0 con 0x7fb04c09caf0 2024-04-25T19:12:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.263+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb04c050740 con 0x7fb04c09caf0 2024-04-25T19:12:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.264+0000 7fb04a7fc640 1 -- 172.21.15.28:0/88662044 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb044018400 con 0x7fb04c09caf0 2024-04-25T19:12:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.265+0000 7fb04a7fc640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb0380744b0 0x7fb0380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.265+0000 7fb04a7fc640 1 -- 172.21.15.28:0/88662044 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb044096410 con 0x7fb04c09caf0 2024-04-25T19:12:06.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.265+0000 7fb051065640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb0380744b0 0x7fb0380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.265+0000 7fb051065640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb0380744b0 0x7fb0380769c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb040034000 tx=0x7fb04002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.269+0000 7fb04a7fc640 1 -- 172.21.15.28:0/88662044 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb044014030 con 0x7fb04c09caf0 2024-04-25T19:12:06.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.494+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb04c08cd40 con 0x7fb04c09caf0 2024-04-25T19:12:06.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.494+0000 7fb04a7fc640 1 -- 172.21.15.28:0/88662044 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fb044060860 con 0x7fb04c09caf0 2024-04-25T19:12:06.496 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T19:12:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb0380744b0 msgr2=0x7fb0380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb0380744b0 0x7fb0380769c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb040034000 tx=0x7fb04002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c09caf0 msgr2=0x7fb04c1dbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c09caf0 0x7fb04c1dbce0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb04400cde0 tx=0x7fb04400a650 comp rx=0 tx=0).stop 2024-04-25T19:12:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 shutdown_connections 2024-04-25T19:12:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb0380744b0 0x7fb0380769c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb04c0d2670 0x7fb04c1e7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb04c0b7710 0x7fb04c06e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 --2- 172.21.15.28:0/88662044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb04c09caf0 0x7fb04c1dbce0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 >> 172.21.15.28:0/88662044 conn(0x7fb04c0d03f0 msgr2=0x7fb04c0714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 shutdown_connections 2024-04-25T19:12:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:06.496+0000 7fb053aca640 1 -- 172.21.15.28:0/88662044 wait complete. 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T19:12:06.509 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T19:12:06.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:06.510 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-04-25T19:12:06.529 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:06.529 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:06.529 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:06.529 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:06.529 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42735sec preferred_lft 42735sec 2024-04-25T19:12:06.530 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-04-25T19:12:06.531 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:06.531 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:06.531 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:06.531 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-04-25T19:12:06.531 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T19:12:06.531 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:06.531 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:06.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add name ceph-brx type bridge 2024-04-25T19:12:06.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr flush dev ceph-brx 2024-04-25T19:12:06.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set ceph-brx up 2024-04-25T19:12:06.531 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T19:12:06.531 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:06.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:06.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:06.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:06.740 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T19:12:06.765 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-04-25T19:12:06.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:06.767 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-04-25T19:12:06.832 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-04-25T19:12:06.832 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T19:12:06.832 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T19:12:06.832 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T19:12:06.832 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T19:12:06.832 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:06.832 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:06.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T19:12:06.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T19:12:06.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T19:12:06.833 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:06.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:07.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:07.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:07.106 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-25T19:12:07.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:07.121 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list-id 2024-04-25T19:12:07.177 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:07.177 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:07.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T19:12:07.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-25T19:12:07.177 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:07.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:07.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:07.296 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-25T19:12:07.296 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:07.296 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:07.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-25T19:12:07.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T19:12:07.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-25T19:12:07.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-25T19:12:07.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-25T19:12:07.296 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:07.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:07.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:07.441 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:07.441 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:07.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-04-25T19:12:07.441 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T19:12:07.441 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:07.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:07.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:07.518 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:07.519 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T19:12:07.534 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T19:12:07.534 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:07.534 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T19:12:07.589 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T19:12:07.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:07.590 DEBUG:teuthology.orchestra.run.smithi053:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:07.688 INFO:teuthology.orchestra.run.smithi053.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:07.688 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:07.790 INFO:teuthology.orchestra.run.smithi053.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:07.791 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:07.791 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.53:3300,172.21.15.53:3301" 2024-04-25T19:12:07.791 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: trying mount with new device syntax: 1@5d8007d4-24a5-41be-aa4d-cd64fedac62a.cephfs=/ 2024-04-25T19:12:07.791 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.28:3300/172.21.15.53:3300/172.21.15.53:3301" will pass to kernel 2024-04-25T19:12:07.792 INFO:teuthology.orchestra.run.smithi053.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T19:12:07.792 INFO:teuthology.orchestra.run.smithi053.stdout: You just mounted a file system that supports labels which does not 2024-04-25T19:12:07.792 INFO:teuthology.orchestra.run.smithi053.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T19:12:07.792 INFO:teuthology.orchestra.run.smithi053.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T19:12:07.792 INFO:teuthology.orchestra.run.smithi053.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T19:12:07.794 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T19:12:07.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:07.794 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T19:12:07.818 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:07.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:07.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:07.915 INFO:teuthology.orchestra.run.smithi053.stdout:client4842 2024-04-25T19:12:07.915 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:12:07.915 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:07.931 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:08.028 INFO:teuthology.orchestra.run.smithi053.stdout:client4842 2024-04-25T19:12:08.028 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:08.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:08.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:08.085 INFO:teuthology.orchestra.run.smithi053.stdout:client4842 2024-04-25T19:12:08.085 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5d8007d4-24a5-41be-aa4d-cd64fedac62a.client4842/status') 2024-04-25T19:12:08.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:08.143 INFO:teuthology.orchestra.run.smithi053.stdout:instance: client.4842 (3)172.21.15.53:0/1719196709 2024-04-25T19:12:08.144 INFO:teuthology.orchestra.run.smithi053.stdout:blocklisted: false 2024-04-25T19:12:08.144 INFO:teuthology.orchestra.run.smithi053.stderr:0+1 records in 2024-04-25T19:12:08.144 INFO:teuthology.orchestra.run.smithi053.stderr:0+1 records out 2024-04-25T19:12:08.144 INFO:teuthology.orchestra.run.smithi053.stderr:70 bytes copied, 2.5158e-05 s, 2.8 MB/s 2024-04-25T19:12:08.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:08.151 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-04-25T19:12:08.151 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T19:12:08.151 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T19:12:08.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T19:12:08.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.275+0000 7f71ab092640 1 Processor -- start 2024-04-25T19:12:08.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.275+0000 7f71ab092640 1 -- start start 2024-04-25T19:12:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71ab092640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f71a40b32d0 0x7f71a40b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71ab092640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:08.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71ab092640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71a409d1a0 con 0x7f71a409cd90 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71ab092640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f71a406df50 con 0x7f71a40b32d0 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71ab092640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f71a40d4870 con 0x7f71a40b3c20 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71a8e2e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f71a409cd90 0x7f71a406e300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38018/0 (socket says 172.21.15.28:38018) 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71a8e2e640 1 -- 172.21.15.28:0/2237782286 learned_addr learned my addr 172.21.15.28:0/2237782286 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71a962f640 1 --2- 172.21.15.28:0/2237782286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71a3fff640 1 --2- 172.21.15.28:0/2237782286 >> v2:172.21.15.53:3300/0 conn(0x7f71a40b32d0 0x7f71a40b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71a37fe640 1 -- 172.21.15.28:0/2237782286 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2058228118 0 0) 0x7f71a409d1a0 con 0x7f71a409cd90 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.277+0000 7f71a37fe640 1 -- 172.21.15.28:0/2237782286 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7194003520 con 0x7f71a409cd90 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.278+0000 7f71a962f640 1 -- 172.21.15.28:0/2237782286 >> v2:172.21.15.53:3300/0 conn(0x7f71a40b32d0 msgr2=0x7f71a40b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.278+0000 7f71a962f640 1 --2- 172.21.15.28:0/2237782286 >> v2:172.21.15.53:3300/0 conn(0x7f71a40b32d0 0x7f71a40b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.278+0000 7f71a962f640 1 -- 172.21.15.28:0/2237782286 >> v1:172.21.15.28:6789/0 conn(0x7f71a409cd90 legacy=0x7f71a406e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.278+0000 7f71a37fe640 1 -- 172.21.15.28:0/2237782286 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1723917692 0 0) 0x7f7194003520 con 0x7f71a409cd90 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.278+0000 7f71a962f640 1 -- 172.21.15.28:0/2237782286 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a406d960 con 0x7f71a40b3c20 2024-04-25T19:12:08.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.278+0000 7f71a3fff640 1 --2- 172.21.15.28:0/2237782286 >> v2:172.21.15.53:3300/0 conn(0x7f71a40b32d0 0x7f71a40b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.278+0000 7f71a962f640 1 --2- 172.21.15.28:0/2237782286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40d1e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f719c009d90 tx=0x7f719c02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa6e55f6d9e6116a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:08.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71a37fe640 1 -- 172.21.15.28:0/2237782286 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198001690 con 0x7f71a40b3c20 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71a37fe640 1 -- 172.21.15.28:0/2237782286 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f719c007540 con 0x7f71a40b3c20 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71a37fe640 1 -- 172.21.15.28:0/2237782286 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f719c0078f0 con 0x7f71a40b3c20 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71ab092640 1 -- 172.21.15.28:0/2237782286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 msgr2=0x7f71a40d1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71ab092640 1 --2- 172.21.15.28:0/2237782286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40d1e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f719c009d90 tx=0x7f719c02a560 comp rx=0 tx=0).stop 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71ab092640 1 -- 172.21.15.28:0/2237782286 shutdown_connections 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71ab092640 1 --2- 172.21.15.28:0/2237782286 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40d1e50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71ab092640 1 --2- 172.21.15.28:0/2237782286 >> v2:172.21.15.53:3300/0 conn(0x7f71a40b32d0 0x7f71a40b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71ab092640 1 -- 172.21.15.28:0/2237782286 >> 172.21.15.28:0/2237782286 conn(0x7f71a40d03f0 msgr2=0x7f71a40d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71ab092640 1 -- 172.21.15.28:0/2237782286 shutdown_connections 2024-04-25T19:12:08.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.279+0000 7f71ab092640 1 -- 172.21.15.28:0/2237782286 wait complete. 2024-04-25T19:12:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.281+0000 7f71ab092640 1 Processor -- start 2024-04-25T19:12:08.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.281+0000 7f71ab092640 1 -- start start 2024-04-25T19:12:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71ab092640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a409cd90 0x7f71a41e31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71ab092640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a40b32d0 0x7f71a40cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71ab092640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71ab092640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71a40b9180 con 0x7f71a409cd90 2024-04-25T19:12:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71ab092640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f71a406dc70 con 0x7f71a40b32d0 2024-04-25T19:12:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71ab092640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f71a406ea10 con 0x7f71a40b3c20 2024-04-25T19:12:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a8e2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a409cd90 0x7f71a41e31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a8e2e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a409cd90 0x7f71a41e31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44962/0 (socket says 172.21.15.28:44962) 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a8e2e640 1 -- 172.21.15.28:0/2036999976 learned_addr learned my addr 172.21.15.28:0/2036999976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a962f640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a3fff640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a40b32d0 0x7f71a40cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a8e2e640 1 -- 172.21.15.28:0/2036999976 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 msgr2=0x7f71a40cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a8e2e640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a8e2e640 1 -- 172.21.15.28:0/2036999976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a40b32d0 msgr2=0x7f71a40cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a8e2e640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a40b32d0 0x7f71a40cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.283+0000 7f71a8e2e640 1 -- 172.21.15.28:0/2036999976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a40caa20 con 0x7f71a409cd90 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.284+0000 7f71a8e2e640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a409cd90 0x7f71a41e31c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f719800c530 tx=0x7f719800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.284+0000 7f71a17fa640 1 -- 172.21.15.28:0/2036999976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7198018070 con 0x7f71a409cd90 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.284+0000 7f71a17fa640 1 -- 172.21.15.28:0/2036999976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71980074e0 con 0x7f71a409cd90 2024-04-25T19:12:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.284+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a40b0ba0 con 0x7f71a409cd90 2024-04-25T19:12:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.284+0000 7f71a17fa640 1 -- 172.21.15.28:0/2036999976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71980049e0 con 0x7f71a409cd90 2024-04-25T19:12:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.284+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a40b1160 con 0x7f71a409cd90 2024-04-25T19:12:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.285+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a40d1a30 con 0x7f71a409cd90 2024-04-25T19:12:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.286+0000 7f71a17fa640 1 -- 172.21.15.28:0/2036999976 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f719801b070 con 0x7f71a409cd90 2024-04-25T19:12:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.286+0000 7f71a17fa640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71880744e0 0x7f71880769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.286+0000 7f71a17fa640 1 -- 172.21.15.28:0/2036999976 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7198094970 con 0x7f71a409cd90 2024-04-25T19:12:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.286+0000 7f71a3fff640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71880744e0 0x7f71880769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.287+0000 7f71a3fff640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71880744e0 0x7f71880769f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f71a41f9c30 tx=0x7f7190009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.291+0000 7f71a17fa640 1 -- 172.21.15.28:0/2036999976 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f71a40d1a30 con 0x7f71a409cd90 2024-04-25T19:12:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.521+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f71a4050740 con 0x7f71a409cd90 2024-04-25T19:12:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.521+0000 7f71a17fa640 1 -- 172.21.15.28:0/2036999976 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f719805e800 con 0x7f71a409cd90 2024-04-25T19:12:08.523 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T19:12:08.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.523+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71880744e0 msgr2=0x7f71880769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.523+0000 7f71ab092640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71880744e0 0x7f71880769f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f71a41f9c30 tx=0x7f7190009190 comp rx=0 tx=0).stop 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.523+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a409cd90 msgr2=0x7f71a41e31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.523+0000 7f71ab092640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a409cd90 0x7f71a41e31c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f719800c530 tx=0x7f719800c9f0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.524+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 shutdown_connections 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.524+0000 7f71ab092640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71880744e0 0x7f71880769f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.524+0000 7f71ab092640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a40b3c20 0x7f71a40cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.524+0000 7f71ab092640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a40b32d0 0x7f71a40cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.524+0000 7f71ab092640 1 --2- 172.21.15.28:0/2036999976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a409cd90 0x7f71a41e31c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.524+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 >> 172.21.15.28:0/2036999976 conn(0x7f71a40d03f0 msgr2=0x7f71a40724a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.524+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 shutdown_connections 2024-04-25T19:12:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:08.524+0000 7f71ab092640 1 -- 172.21.15.28:0/2036999976 wait complete. 2024-04-25T19:12:08.536 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T19:12:08.537 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T19:12:08.537 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-04-25T19:12:08.537 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-25T19:12:08.537 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T19:12:08.537 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T19:12:08.537 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T19:12:08.537 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T19:12:08.537 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T19:12:08.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:08.537 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-04-25T19:12:08.554 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:08.554 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:08.554 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:08.554 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:08.554 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:08.554 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:08.554 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42733sec preferred_lft 42733sec 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:08.555 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 9a:7e:57:04:62:86 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::90b0:64ff:fe80:b822/64 scope link 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 9a:7e:57:04:62:86 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::987e:57ff:fe04:6286/64 scope link tentative 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:08.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:08.556 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T19:12:08.619 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-04-25T19:12:08.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:08.620 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-04-25T19:12:08.677 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-04-25T19:12:08.678 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T19:12:08.678 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T19:12:08.678 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T19:12:08.678 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T19:12:08.678 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:08.678 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:08.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T19:12:08.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T19:12:08.678 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T19:12:08.679 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:08.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:08.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:08.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:08.805 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-25T19:12:08.820 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T19:12:08.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:08.821 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list-id 2024-04-25T19:12:08.875 INFO:teuthology.orchestra.run.smithi053.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-25T19:12:08.876 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:08.876 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:08.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T19:12:08.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-25T19:12:08.876 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:08.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:08.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:08.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:08.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T19:12:09.018 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:09.018 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:09.019 INFO:teuthology.orchestra.run.smithi053.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T19:12:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 22:ba:23:80:01:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T19:12:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T19:12:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::20ba:23ff:fe80:163/64 scope link tentative 2024-04-25T19:12:09.019 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:09.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:09.020 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T19:12:09.087 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:09.087 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:09.087 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:09.087 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:09.087 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:09.087 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:09.087 INFO:teuthology.orchestra.run.smithi053.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T19:12:09.087 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 22:ba:23:80:01:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T19:12:09.088 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T19:12:09.088 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:09.088 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::20ba:23ff:fe80:163/64 scope link tentative 2024-04-25T19:12:09.088 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:09.089 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-25T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-25T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-25T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-25T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-25T19:12:09.089 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:09.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:09.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:09.241 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:09.241 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-04-25T19:12:09.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.1 up 2024-04-25T19:12:09.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T19:12:09.241 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-04-25T19:12:09.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:09.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:09.317 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:09.318 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-25T19:12:09.345 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-25T19:12:09.345 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:09.345 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-25T19:12:09.400 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-04-25T19:12:09.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:09.400 DEBUG:teuthology.orchestra.run.smithi053:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:09.494 INFO:teuthology.orchestra.run.smithi053.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:09.494 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:09.503 INFO:teuthology.orchestra.run.smithi053.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:09.503 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:09.503 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.53:3300,172.21.15.53:3301" 2024-04-25T19:12:09.503 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: trying mount with new device syntax: 3@5d8007d4-24a5-41be-aa4d-cd64fedac62a.cephfs=/ 2024-04-25T19:12:09.503 INFO:teuthology.orchestra.run.smithi053.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.28:3300/172.21.15.53:3300/172.21.15.53:3301" will pass to kernel 2024-04-25T19:12:09.504 INFO:teuthology.orchestra.run.smithi053.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-04-25T19:12:09.504 INFO:teuthology.orchestra.run.smithi053.stdout: You just mounted a file system that supports labels which does not 2024-04-25T19:12:09.504 INFO:teuthology.orchestra.run.smithi053.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T19:12:09.504 INFO:teuthology.orchestra.run.smithi053.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T19:12:09.505 INFO:teuthology.orchestra.run.smithi053.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T19:12:09.505 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T19:12:09.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:09.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-25T19:12:09.535 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:09.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:09.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:09.631 INFO:teuthology.orchestra.run.smithi053.stdout:client4565 2024-04-25T19:12:09.632 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:12:09.632 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:09.648 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:09.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:09.743 INFO:teuthology.orchestra.run.smithi053.stdout:client4565 2024-04-25T19:12:09.744 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:09.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:09.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:09.799 INFO:teuthology.orchestra.run.smithi053.stdout:client4565 2024-04-25T19:12:09.800 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/5d8007d4-24a5-41be-aa4d-cd64fedac62a.client4565/status') 2024-04-25T19:12:09.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:09.858 INFO:teuthology.orchestra.run.smithi053.stdout:instance: client.4565 (3)192.168.0.2:0/594430437 2024-04-25T19:12:09.858 INFO:teuthology.orchestra.run.smithi053.stdout:blocklisted: false 2024-04-25T19:12:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:0+1 records in 2024-04-25T19:12:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:0+1 records out 2024-04-25T19:12:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:68 bytes copied, 2.451e-05 s, 2.8 MB/s 2024-04-25T19:12:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-04-25T19:12:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:09.866 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-25T19:12:09.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:10.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.008+0000 7f3365c42640 1 Processor -- start 2024-04-25T19:12:10.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.008+0000 7f3365c42640 1 -- start start 2024-04-25T19:12:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f3365c42640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33600b5130 0x7f3360161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f3365c42640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f336006ded0 0x7f336006e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f3365c42640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f336006e820 con 0x7f33600d0830 2024-04-25T19:12:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f3365c42640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f3360163c30 con 0x7f336006ded0 2024-04-25T19:12:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f3365c42640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f3360164a90 con 0x7f33600b5130 2024-04-25T19:12:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335effd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f336006ded0 0x7f336006e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f33600d0830 0x7f33600d0c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38022/0 (socket says 172.21.15.28:38022) 2024-04-25T19:12:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335ffff640 1 -- 172.21.15.28:0/2608666899 learned_addr learned my addr 172.21.15.28:0/2608666899 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335f7fe640 1 --2- 172.21.15.28:0/2608666899 >> v2:172.21.15.53:3300/0 conn(0x7f33600b5130 0x7f3360161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335effd640 1 -- 172.21.15.28:0/2608666899 >> v2:172.21.15.53:3300/0 conn(0x7f33600b5130 msgr2=0x7f3360161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335effd640 1 --2- 172.21.15.28:0/2608666899 >> v2:172.21.15.53:3300/0 conn(0x7f33600b5130 0x7f3360161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335effd640 1 -- 172.21.15.28:0/2608666899 >> v1:172.21.15.28:6789/0 conn(0x7f33600d0830 legacy=0x7f33600d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335effd640 1 -- 172.21.15.28:0/2608666899 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3360163e80 con 0x7f336006ded0 2024-04-25T19:12:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.010+0000 7f335e7fc640 1 -- 172.21.15.28:0/2608666899 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 583448614 0 0) 0x7f3358001490 con 0x7f33600d0830 2024-04-25T19:12:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.011+0000 7f335effd640 1 --2- 172.21.15.28:0/2608666899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f336006e2e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f336006ec40 tx=0x7f335402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7b4e78c6ccb41dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.011+0000 7f335e7fc640 1 -- 172.21.15.28:0/2608666899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354037070 con 0x7f336006ded0 2024-04-25T19:12:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.011+0000 7f335e7fc640 1 -- 172.21.15.28:0/2608666899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3354032440 con 0x7f336006ded0 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.011+0000 7f335e7fc640 1 -- 172.21.15.28:0/2608666899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354032810 con 0x7f336006ded0 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.012+0000 7f3365c42640 1 -- 172.21.15.28:0/2608666899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 msgr2=0x7f336006e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.012+0000 7f3365c42640 1 --2- 172.21.15.28:0/2608666899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f336006e2e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f336006ec40 tx=0x7f335402a1a0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.012+0000 7f3365c42640 1 -- 172.21.15.28:0/2608666899 shutdown_connections 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.012+0000 7f3365c42640 1 --2- 172.21.15.28:0/2608666899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f336006e2e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.012+0000 7f3365c42640 1 --2- 172.21.15.28:0/2608666899 >> v2:172.21.15.53:3300/0 conn(0x7f33600b5130 0x7f3360161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.012+0000 7f3365c42640 1 -- 172.21.15.28:0/2608666899 >> 172.21.15.28:0/2608666899 conn(0x7f33600b7ad0 msgr2=0x7f33600b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.012+0000 7f3365c42640 1 -- 172.21.15.28:0/2608666899 shutdown_connections 2024-04-25T19:12:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.012+0000 7f3365c42640 1 -- 172.21.15.28:0/2608666899 wait complete. 2024-04-25T19:12:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.014+0000 7f3365c42640 1 Processor -- start 2024-04-25T19:12:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.014+0000 7f3365c42640 1 -- start start 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.015+0000 7f3365c42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f3360152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.015+0000 7f3365c42640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33600b5130 0x7f33600d49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.015+0000 7f3365c42640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33600d0830 0x7f33600d3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.015+0000 7f3365c42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3360164720 con 0x7f336006ded0 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.015+0000 7f3365c42640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3360164440 con 0x7f33600d0830 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.015+0000 7f3365c42640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f33601607e0 con 0x7f33600b5130 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f3360152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f3360152700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44986/0 (socket says 172.21.15.28:44986) 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 -- 172.21.15.28:0/3243770954 learned_addr learned my addr 172.21.15.28:0/3243770954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335ffff640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33600d0830 0x7f33600d3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335effd640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33600b5130 0x7f33600d49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 -- 172.21.15.28:0/3243770954 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33600b5130 msgr2=0x7f33600d49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33600b5130 0x7f33600d49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 -- 172.21.15.28:0/3243770954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33600d0830 msgr2=0x7f33600d3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33600d0830 0x7f33600d3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 -- 172.21.15.28:0/3243770954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3360152c40 con 0x7f336006ded0 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f335f7fe640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f3360152700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f335000e870 tx=0x7f335000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f3347fff640 1 -- 172.21.15.28:0/3243770954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3350017070 con 0x7f336006ded0 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f3347fff640 1 -- 172.21.15.28:0/3243770954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335000c940 con 0x7f336006ded0 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f3347fff640 1 -- 172.21.15.28:0/3243770954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335000cd10 con 0x7f336006ded0 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336015bd20 con 0x7f336006ded0 2024-04-25T19:12:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.016+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f336015c2e0 con 0x7f336006ded0 2024-04-25T19:12:10.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.018+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3360050740 con 0x7f336006ded0 2024-04-25T19:12:10.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.018+0000 7f3347fff640 1 -- 172.21.15.28:0/3243770954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f335001a070 con 0x7f336006ded0 2024-04-25T19:12:10.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.019+0000 7f3347fff640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f334c0744b0 0x7f334c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.019+0000 7f3347fff640 1 -- 172.21.15.28:0/3243770954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6735+0+0 (secure 0 0 0) 0x7f33500957b0 con 0x7f336006ded0 2024-04-25T19:12:10.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.019+0000 7f335effd640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f334c0744b0 0x7f334c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.019+0000 7f335effd640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f334c0744b0 0x7f334c0769c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3354034000 tx=0x7f335402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.024+0000 7f3347fff640 1 -- 172.21.15.28:0/3243770954 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f335005f4f0 con 0x7f336006ded0 2024-04-25T19:12:10.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.221+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33600d2fe0 con 0x7f336006ded0 2024-04-25T19:12:10.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.222+0000 7f3347fff640 1 -- 172.21.15.28:0/3243770954 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f33500637f0 con 0x7f336006ded0 2024-04-25T19:12:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:10.225 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f334c0744b0 msgr2=0x7f334c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f334c0744b0 0x7f334c0769c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3354034000 tx=0x7f335402e000 comp rx=0 tx=0).stop 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 msgr2=0x7f3360152700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f3360152700 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f335000e870 tx=0x7f335000ed30 comp rx=0 tx=0).stop 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 shutdown_connections 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f334c0744b0 0x7f334c0769c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33600d0830 0x7f33600d3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f33600b5130 0x7f33600d49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 --2- 172.21.15.28:0/3243770954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f336006ded0 0x7f3360152700 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.224+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 >> 172.21.15.28:0/3243770954 conn(0x7f33600b7ad0 msgr2=0x7f336015f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:10.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.225+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 shutdown_connections 2024-04-25T19:12:10.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.225+0000 7f3365c42640 1 -- 172.21.15.28:0/3243770954 wait complete. 2024-04-25T19:12:10.238 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:10.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.352+0000 7f47c6f3f640 1 Processor -- start 2024-04-25T19:12:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.352+0000 7f47c6f3f640 1 -- start start 2024-04-25T19:12:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.353+0000 7f47c6f3f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f47c009caf0 0x7f47c00d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.353+0000 7f47c6f3f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47c009cf00 0x7f47c00ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.353+0000 7f47c6f3f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47c00d1ce0 con 0x7f47c00d2670 2024-04-25T19:12:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.353+0000 7f47c6f3f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f47c00b8fa0 con 0x7f47c009caf0 2024-04-25T19:12:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.353+0000 7f47c6f3f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f47c00cf8d0 con 0x7f47c009cf00 2024-04-25T19:12:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.353+0000 7f47bffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f47c00d2670 0x7f47c00b71d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38038/0 (socket says 172.21.15.28:38038) 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.353+0000 7f47bffff640 1 -- 172.21.15.28:0/416265174 learned_addr learned my addr 172.21.15.28:0/416265174 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47c4cdb640 1 --2- 172.21.15.28:0/416265174 >> v2:172.21.15.53:3300/0 conn(0x7f47c009caf0 0x7f47c00d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47c54dc640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47c009cf00 0x7f47c00ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3061368800 0 0) 0x7f47c00d1ce0 con 0x7f47c00d2670 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47ac003520 con 0x7f47c00d2670 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 938285735 0 0) 0x7f47ac003520 con 0x7f47c00d2670 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47c009cf00 msgr2=0x7f47c00ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47c009cf00 0x7f47c00ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 >> v2:172.21.15.53:3300/0 conn(0x7f47c009caf0 msgr2=0x7f47c00d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 --2- 172.21.15.28:0/416265174 >> v2:172.21.15.53:3300/0 conn(0x7f47c009caf0 0x7f47c00d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c00cfbb0 con 0x7f47c00d2670 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f47a8001670 con 0x7f47c00d2670 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 >> v1:172.21.15.28:6789/0 conn(0x7f47c00d2670 legacy=0x7f47c00b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47c009cf00 0x7f47ac0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47ac003f30 0x7f47ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47ac004d40 0x7f47ac005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00b8fa0 con 0x7f47ac003f30 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00cf8d0 con 0x7f47ac004d40 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f47ac00bac0 con 0x7f47c009cf00 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bffff640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47ac003f30 0x7f47ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47c4cdb640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47c009cf00 0x7f47ac0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47c54dc640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47ac004d40 0x7f47ac005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bffff640 1 -- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47c009cf00 msgr2=0x7f47ac0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bffff640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47c009cf00 0x7f47ac0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bffff640 1 -- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47ac004d40 msgr2=0x7f47ac005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bffff640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47ac004d40 0x7f47ac005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.354+0000 7f47bffff640 1 -- 172.21.15.28:0/416265174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47ac003520 con 0x7f47ac003f30 2024-04-25T19:12:10.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.355+0000 7f47bffff640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47ac003f30 0x7f47ac004800 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f47a8001050 tx=0x7f47ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35aa14378b82f8e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.355+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a802e070 con 0x7f47ac003f30 2024-04-25T19:12:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.355+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a8005af0 con 0x7f47ac003f30 2024-04-25T19:12:10.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.355+0000 7f47bf7fe640 1 -- 172.21.15.28:0/416265174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8007d70 con 0x7f47ac003f30 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.356+0000 7f47c6f3f640 1 -- 172.21.15.28:0/416265174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47ac003f30 msgr2=0x7f47ac004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.356+0000 7f47c6f3f640 1 --2- 172.21.15.28:0/416265174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47ac003f30 0x7f47ac004800 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f47a8001050 tx=0x7f47ac005830 comp rx=0 tx=0).stop 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.356+0000 7f47c54dc640 1 -- 172.21.15.28:0/416265174 reap_dead start 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.356+0000 7f47c6f3f640 1 -- 172.21.15.28:0/416265174 shutdown_connections 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.356+0000 7f47c6f3f640 1 -- 172.21.15.28:0/416265174 >> 172.21.15.28:0/416265174 conn(0x7f47c00d03f0 msgr2=0x7f47c00d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.356+0000 7f47c6f3f640 1 -- 172.21.15.28:0/416265174 shutdown_connections 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.356+0000 7f47c6f3f640 1 -- 172.21.15.28:0/416265174 wait complete. 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.358+0000 7f47c6f3f640 1 Processor -- start 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.358+0000 7f47c6f3f640 1 -- start start 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.359+0000 7f47c6f3f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47ac003f30 0x7f47c00b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.359+0000 7f47c6f3f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47c01e61b0 0x7f47c00c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.359+0000 7f47c6f3f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c00c9b90 0x7f47c00b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.359+0000 7f47c6f3f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00b9430 con 0x7f47c01e61b0 2024-04-25T19:12:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.359+0000 7f47c6f3f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f47c00b78e0 con 0x7f47c00c9b90 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.359+0000 7f47c6f3f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f47c00d4790 con 0x7f47ac003f30 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.359+0000 7f47bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47c01e61b0 0x7f47c00c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47c01e61b0 0x7f47c00c9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45004/0 (socket says 172.21.15.28:45004) 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bffff640 1 -- 172.21.15.28:0/730932692 learned_addr learned my addr 172.21.15.28:0/730932692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47c4cdb640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47ac003f30 0x7f47c00b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47c54dc640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c00c9b90 0x7f47c00b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bffff640 1 -- 172.21.15.28:0/730932692 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47ac003f30 msgr2=0x7f47c00b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bffff640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47ac003f30 0x7f47c00b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bffff640 1 -- 172.21.15.28:0/730932692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c00c9b90 msgr2=0x7f47c00b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bffff640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c00c9b90 0x7f47c00b05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bffff640 1 -- 172.21.15.28:0/730932692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47c01e6d10 con 0x7f47c01e61b0 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bffff640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47c01e61b0 0x7f47c00c9650 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f47a8034040 tx=0x7f47a8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bd7fa640 1 -- 172.21.15.28:0/730932692 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8042070 con 0x7f47c01e61b0 2024-04-25T19:12:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bd7fa640 1 -- 172.21.15.28:0/730932692 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47a8007480 con 0x7f47c01e61b0 2024-04-25T19:12:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47c00c9fa0 con 0x7f47c01e61b0 2024-04-25T19:12:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47bd7fa640 1 -- 172.21.15.28:0/730932692 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47a8007d50 con 0x7f47c01e61b0 2024-04-25T19:12:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.360+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47c00b4490 con 0x7f47c01e61b0 2024-04-25T19:12:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.361+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47c00b2290 con 0x7f47c01e61b0 2024-04-25T19:12:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.362+0000 7f47bd7fa640 1 -- 172.21.15.28:0/730932692 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f47a8045070 con 0x7f47c01e61b0 2024-04-25T19:12:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.363+0000 7f47bd7fa640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47a00744e0 0x7f47a00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.363+0000 7f47bd7fa640 1 -- 172.21.15.28:0/730932692 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f47a80b5f70 con 0x7f47c01e61b0 2024-04-25T19:12:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.363+0000 7f47c4cdb640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47a00744e0 0x7f47a00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.363+0000 7f47c4cdb640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47a00744e0 0x7f47a00769f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f47c0069f40 tx=0x7f47b800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.367+0000 7f47bd7fa640 1 -- 172.21.15.28:0/730932692 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47c00c9fa0 con 0x7f47c01e61b0 2024-04-25T19:12:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.553+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f47c00b3200 con 0x7f47c01e61b0 2024-04-25T19:12:10.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.553+0000 7f47bd7fa640 1 -- 172.21.15.28:0/730932692 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f47c01e6d10 con 0x7f47c01e61b0 2024-04-25T19:12:10.555 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47a00744e0 msgr2=0x7f47a00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47a00744e0 0x7f47a00769f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f47c0069f40 tx=0x7f47b800b040 comp rx=0 tx=0).stop 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47c01e61b0 msgr2=0x7f47c00c9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47c01e61b0 0x7f47c00c9650 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f47a8034040 tx=0x7f47a8000fe0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 shutdown_connections 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47a00744e0 0x7f47a00769f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c00c9b90 0x7f47c00b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47c01e61b0 0x7f47c00c9650 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 --2- 172.21.15.28:0/730932692 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f47ac003f30 0x7f47c00b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 >> 172.21.15.28:0/730932692 conn(0x7f47c00d03f0 msgr2=0x7f47c0072240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 shutdown_connections 2024-04-25T19:12:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.555+0000 7f47c6f3f640 1 -- 172.21.15.28:0/730932692 wait complete. 2024-04-25T19:12:10.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:10.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.682+0000 7f0e5d9eb640 1 Processor -- start 2024-04-25T19:12:10.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.682+0000 7f0e5d9eb640 1 -- start start 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.683+0000 7f0e5d9eb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e580b7fd0 0x7f0e58160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.683+0000 7f0e5d9eb640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0e58164800 0x7f0e580b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.683+0000 7f0e5d9eb640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e580d15b0 con 0x7f0e580b4c80 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.683+0000 7f0e5d9eb640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f0e58164c10 con 0x7f0e580b7fd0 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.683+0000 7f0e5d9eb640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f0e580d12d0 con 0x7f0e58164800 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.683+0000 7f0e567fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e580b7fd0 0x7f0e58160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.683+0000 7f0e56ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0e580b4c80 0x7f0e580b78c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50470/0 (socket says 172.21.15.28:50470) 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.683+0000 7f0e56ffd640 1 -- 172.21.15.28:0/824767651 learned_addr learned my addr 172.21.15.28:0/824767651 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.684+0000 7f0e577fe640 1 --2- 172.21.15.28:0/824767651 >> v2:172.21.15.53:3300/0 conn(0x7f0e58164800 0x7f0e580b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.684+0000 7f0e567fc640 1 -- 172.21.15.28:0/824767651 >> v2:172.21.15.53:3300/0 conn(0x7f0e58164800 msgr2=0x7f0e580b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.684+0000 7f0e567fc640 1 --2- 172.21.15.28:0/824767651 >> v2:172.21.15.53:3300/0 conn(0x7f0e58164800 0x7f0e580b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.684+0000 7f0e567fc640 1 -- 172.21.15.28:0/824767651 >> v1:172.21.15.28:6789/0 conn(0x7f0e580b4c80 legacy=0x7f0e580b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.684+0000 7f0e567fc640 1 -- 172.21.15.28:0/824767651 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e58163d20 con 0x7f0e580b7fd0 2024-04-25T19:12:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.684+0000 7f0e55ffb640 1 -- 172.21.15.28:0/824767651 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1626731809 0 0) 0x7f0e580d15b0 con 0x7f0e580b4c80 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.684+0000 7f0e567fc640 1 --2- 172.21.15.28:0/824767651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e580b7fd0 0x7f0e58160220 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0e580d56e0 tx=0x7f0e4c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63152424759ccdee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e55ffb640 1 -- 172.21.15.28:0/824767651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c037070 con 0x7f0e580b7fd0 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e55ffb640 1 -- 172.21.15.28:0/824767651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e4c032440 con 0x7f0e580b7fd0 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e55ffb640 1 -- 172.21.15.28:0/824767651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c032810 con 0x7f0e580b7fd0 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/824767651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e580b7fd0 msgr2=0x7f0e58160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/824767651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e580b7fd0 0x7f0e58160220 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0e580d56e0 tx=0x7f0e4c02a1a0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/824767651 shutdown_connections 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/824767651 >> v2:172.21.15.53:3300/0 conn(0x7f0e58164800 0x7f0e580b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/824767651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e580b7fd0 0x7f0e58160220 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/824767651 >> 172.21.15.28:0/824767651 conn(0x7f0e580d2980 msgr2=0x7f0e580d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/824767651 shutdown_connections 2024-04-25T19:12:10.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.685+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/824767651 wait complete. 2024-04-25T19:12:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.687+0000 7f0e5d9eb640 1 Processor -- start 2024-04-25T19:12:10.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.687+0000 7f0e5d9eb640 1 -- start start 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.688+0000 7f0e5d9eb640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e580b4c80 0x7f0e58154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e5d9eb640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e58164800 0x7f0e58162400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e5d9eb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5815e8f0 0x7f0e5815ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e5d9eb640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e580b5240 con 0x7f0e5815e8f0 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e5d9eb640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e581642e0 con 0x7f0e58164800 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e5d9eb640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0e580b53c0 con 0x7f0e580b4c80 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5815e8f0 0x7f0e5815ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5815e8f0 0x7f0e5815ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56044/0 (socket says 172.21.15.28:56044) 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 -- 172.21.15.28:0/340773612 learned_addr learned my addr 172.21.15.28:0/340773612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e567fc640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e58164800 0x7f0e58162400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e56ffd640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e580b4c80 0x7f0e58154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 -- 172.21.15.28:0/340773612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e580b4c80 msgr2=0x7f0e58154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e580b4c80 0x7f0e58154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 -- 172.21.15.28:0/340773612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e58164800 msgr2=0x7f0e58162400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e58164800 0x7f0e58162400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 -- 172.21.15.28:0/340773612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5815f240 con 0x7f0e5815e8f0 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e577fe640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5815e8f0 0x7f0e5815ed00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0e48007e60 tx=0x7f0e4800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e3ffff640 1 -- 172.21.15.28:0/340773612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48014420 con 0x7f0e5815e8f0 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e3ffff640 1 -- 172.21.15.28:0/340773612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e48019070 con 0x7f0e5815e8f0 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.689+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e58154680 con 0x7f0e5815e8f0 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.690+0000 7f0e3ffff640 1 -- 172.21.15.28:0/340773612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e48014820 con 0x7f0e5815e8f0 2024-04-25T19:12:10.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.690+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e580b9770 con 0x7f0e5815e8f0 2024-04-25T19:12:10.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.690+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e580ba110 con 0x7f0e5815e8f0 2024-04-25T19:12:10.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.691+0000 7f0e3ffff640 1 -- 172.21.15.28:0/340773612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0e480149c0 con 0x7f0e5815e8f0 2024-04-25T19:12:10.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.692+0000 7f0e3ffff640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0e440744e0 0x7f0e440769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:10.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.692+0000 7f0e3ffff640 1 -- 172.21.15.28:0/340773612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0e48095c80 con 0x7f0e5815e8f0 2024-04-25T19:12:10.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.692+0000 7f0e56ffd640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0e440744e0 0x7f0e440769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:10.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.692+0000 7f0e56ffd640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0e440744e0 0x7f0e440769f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0e50007fd0 tx=0x7f0e50007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:10.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.696+0000 7f0e3ffff640 1 -- 172.21.15.28:0/340773612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e580ba110 con 0x7f0e5815e8f0 2024-04-25T19:12:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.881+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0e580b91f0 con 0x7f0e5815e8f0 2024-04-25T19:12:10.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.882+0000 7f0e3ffff640 1 -- 172.21.15.28:0/340773612 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f0e5815f240 con 0x7f0e5815e8f0 2024-04-25T19:12:10.884 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.883+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0e440744e0 msgr2=0x7f0e440769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.883+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0e440744e0 0x7f0e440769f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f0e50007fd0 tx=0x7f0e50007de0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.883+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5815e8f0 msgr2=0x7f0e5815ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.883+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5815e8f0 0x7f0e5815ed00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0e48007e60 tx=0x7f0e4800e4b0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.883+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 shutdown_connections 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.883+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0e440744e0 0x7f0e440769f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.884+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e5815e8f0 0x7f0e5815ed00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.884+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e58164800 0x7f0e58162400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.884+0000 7f0e5d9eb640 1 --2- 172.21.15.28:0/340773612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e580b4c80 0x7f0e58154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:10.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.884+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 >> 172.21.15.28:0/340773612 conn(0x7f0e580d2980 msgr2=0x7f0e5815a530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:10.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.884+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 shutdown_connections 2024-04-25T19:12:10.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:10.884+0000 7f0e5d9eb640 1 -- 172.21.15.28:0/340773612 wait complete. 2024-04-25T19:12:10.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:11.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.010+0000 7f858c884640 1 Processor -- start 2024-04-25T19:12:11.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.010+0000 7f858c884640 1 -- start start 2024-04-25T19:12:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f858c884640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 0x7f858815e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f858c884640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85880b21d0 0x7f8588157570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f858c884640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f858809b540 0x7f858809b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f858c884640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f858809be90 con 0x7f85880b21d0 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f858c884640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f85880cffe0 con 0x7f858809b540 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f858c884640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f85880d02c0 con 0x7f85880cfa00 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f8585d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85880b21d0 0x7f8588157570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f8585d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85880b21d0 0x7f8588157570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56054/0 (socket says 172.21.15.28:56054) 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.011+0000 7f8585d9b640 1 -- 172.21.15.28:0/4275509810 learned_addr learned my addr 172.21.15.28:0/4275509810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f8586d9d640 1 --2- 172.21.15.28:0/4275509810 >> v2:172.21.15.53:3300/0 conn(0x7f858809b540 0x7f858809b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f858659c640 1 --2- 172.21.15.28:0/4275509810 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 0x7f858815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f8585d9b640 1 -- 172.21.15.28:0/4275509810 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 msgr2=0x7f858815e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f8585d9b640 1 --2- 172.21.15.28:0/4275509810 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 0x7f858815e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f8585d9b640 1 -- 172.21.15.28:0/4275509810 >> v2:172.21.15.53:3300/0 conn(0x7f858809b540 msgr2=0x7f858809b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f8585d9b640 1 --2- 172.21.15.28:0/4275509810 >> v2:172.21.15.53:3300/0 conn(0x7f858809b540 0x7f858809b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f8585d9b640 1 -- 172.21.15.28:0/4275509810 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858815da20 con 0x7f85880b21d0 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f8585d9b640 1 --2- 172.21.15.28:0/4275509810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880b21d0 0x7f8588157570 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8588072aa0 tx=0x7f857002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6878bf9b7cf5a8db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.012+0000 7f858559a640 1 -- 172.21.15.28:0/4275509810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570037070 con 0x7f85880b21d0 2024-04-25T19:12:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858559a640 1 -- 172.21.15.28:0/4275509810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8570032440 con 0x7f85880b21d0 2024-04-25T19:12:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858559a640 1 -- 172.21.15.28:0/4275509810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8570032810 con 0x7f85880b21d0 2024-04-25T19:12:11.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 -- 172.21.15.28:0/4275509810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880b21d0 msgr2=0x7f8588157570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 --2- 172.21.15.28:0/4275509810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880b21d0 0x7f8588157570 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8588072aa0 tx=0x7f857002a140 comp rx=0 tx=0).stop 2024-04-25T19:12:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 -- 172.21.15.28:0/4275509810 shutdown_connections 2024-04-25T19:12:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 --2- 172.21.15.28:0/4275509810 >> v2:172.21.15.53:3300/0 conn(0x7f858809b540 0x7f858809b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 --2- 172.21.15.28:0/4275509810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85880b21d0 0x7f8588157570 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 --2- 172.21.15.28:0/4275509810 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 0x7f858815e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 -- 172.21.15.28:0/4275509810 >> 172.21.15.28:0/4275509810 conn(0x7f85880b7620 msgr2=0x7f85880b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 -- 172.21.15.28:0/4275509810 shutdown_connections 2024-04-25T19:12:11.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.013+0000 7f858c884640 1 -- 172.21.15.28:0/4275509810 wait complete. 2024-04-25T19:12:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.015+0000 7f858c884640 1 Processor -- start 2024-04-25T19:12:11.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.015+0000 7f858c884640 1 -- start start 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.016+0000 7f858c884640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f858809b540 0x7f85881f9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.016+0000 7f858c884640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85880b21d0 0x7f858809c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858c884640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 0x7f85881484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858c884640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f858815d800 con 0x7f858809b540 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858c884640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f858815d680 con 0x7f85880b21d0 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858c884640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f85880b25e0 con 0x7f85880cfa00 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f858809b540 0x7f85881f9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f858809b540 0x7f85881f9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56066/0 (socket says 172.21.15.28:56066) 2024-04-25T19:12:11.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 -- 172.21.15.28:0/3099656740 learned_addr learned my addr 172.21.15.28:0/3099656740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f8585d9b640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85880b21d0 0x7f858809c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f8586d9d640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 0x7f85881484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 -- 172.21.15.28:0/3099656740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 msgr2=0x7f85881484f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 0x7f85881484f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 -- 172.21.15.28:0/3099656740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85880b21d0 msgr2=0x7f858809c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85880b21d0 0x7f858809c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 -- 172.21.15.28:0/3099656740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8588148a30 con 0x7f858809b540 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858659c640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f858809b540 0x7f85881f9a50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8580007e90 tx=0x7f858000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f857effd640 1 -- 172.21.15.28:0/3099656740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f858001a070 con 0x7f858809b540 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f857effd640 1 -- 172.21.15.28:0/3099656740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8580003990 con 0x7f858809b540 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85880d2500 con 0x7f858809b540 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f857effd640 1 -- 172.21.15.28:0/3099656740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8580003d60 con 0x7f858809b540 2024-04-25T19:12:11.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.017+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85881fa580 con 0x7f858809b540 2024-04-25T19:12:11.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.019+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f858806eab0 con 0x7f858809b540 2024-04-25T19:12:11.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.019+0000 7f857effd640 1 -- 172.21.15.28:0/3099656740 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f858001d070 con 0x7f858809b540 2024-04-25T19:12:11.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.020+0000 7f857effd640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85680744b0 0x7f85680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.020+0000 7f857effd640 1 -- 172.21.15.28:0/3099656740 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8580095680 con 0x7f858809b540 2024-04-25T19:12:11.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.020+0000 7f8585d9b640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85680744b0 0x7f85680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.020+0000 7f8585d9b640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85680744b0 0x7f85680769c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8570034000 tx=0x7f857002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.024+0000 7f857effd640 1 -- 172.21.15.28:0/3099656740 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f858005f450 con 0x7f858809b540 2024-04-25T19:12:11.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.213+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8588050740 con 0x7f858809b540 2024-04-25T19:12:11.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.213+0000 7f857effd640 1 -- 172.21.15.28:0/3099656740 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f8580063750 con 0x7f858809b540 2024-04-25T19:12:11.215 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.215+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85680744b0 msgr2=0x7f85680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.215+0000 7f858c884640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85680744b0 0x7f85680769c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8570034000 tx=0x7f857002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f858809b540 msgr2=0x7f85881f9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f858809b540 0x7f85881f9a50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8580007e90 tx=0x7f858000e4b0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 shutdown_connections 2024-04-25T19:12:11.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85680744b0 0x7f85680769c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85880cfa00 0x7f85881484f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85880b21d0 0x7f858809c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 --2- 172.21.15.28:0/3099656740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f858809b540 0x7f85881f9a50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 >> 172.21.15.28:0/3099656740 conn(0x7f85880b7620 msgr2=0x7f858815e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 shutdown_connections 2024-04-25T19:12:11.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.216+0000 7f858c884640 1 -- 172.21.15.28:0/3099656740 wait complete. 2024-04-25T19:12:11.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:11.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.342+0000 7f244bdce640 1 Processor -- start 2024-04-25T19:12:11.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.343+0000 7f244bdce640 1 -- start start 2024-04-25T19:12:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f244bdce640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f244406ed00 0x7f24440d3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f244bdce640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f24440ccf20 0x7f24440cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f244bdce640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2444161990 con 0x7f24440cd870 2024-04-25T19:12:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f244bdce640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f24440d2dc0 con 0x7f24440ccf20 2024-04-25T19:12:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f244bdce640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f24440d24b0 con 0x7f244406ed00 2024-04-25T19:12:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f244a36b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f24440cd870 0x7f24440b3ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50482/0 (socket says 172.21.15.28:50482) 2024-04-25T19:12:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f244a36b640 1 -- 172.21.15.28:0/2507684662 learned_addr learned my addr 172.21.15.28:0/2507684662 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 390597473 0 0) 0x7f2444161990 con 0x7f24440cd870 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f2449369640 1 --2- 172.21.15.28:0/2507684662 >> v2:172.21.15.53:3300/0 conn(0x7f24440ccf20 0x7f24440cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f2449b6a640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f244406ed00 0x7f24440d3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.344+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f242c003520 con 0x7f24440cd870 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3284953730 0 0) 0x7f242c003520 con 0x7f24440cd870 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f244406ed00 msgr2=0x7f24440d3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f244406ed00 0x7f24440d3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 >> v2:172.21.15.53:3300/0 conn(0x7f24440ccf20 msgr2=0x7f24440cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 --2- 172.21.15.28:0/2507684662 >> v2:172.21.15.53:3300/0 conn(0x7f24440ccf20 0x7f24440cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24440d2770 con 0x7f24440cd870 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f2434001a10 con 0x7f24440cd870 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 >> v1:172.21.15.28:6789/0 conn(0x7f24440cd870 legacy=0x7f24440b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f242c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f244406ed00 0x7f242c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c004d40 0x7f242c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24440d2dc0 con 0x7f242c0039b0 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f24440d24b0 con 0x7f242c004d40 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f242c00b9c0 con 0x7f244406ed00 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2449b6a640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f242c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f244a36b640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c004d40 0x7f242c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2449369640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f244406ed00 0x7f242c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2449b6a640 1 -- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f244406ed00 msgr2=0x7f242c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2449b6a640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f244406ed00 0x7f242c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2449b6a640 1 -- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c004d40 msgr2=0x7f242c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2449b6a640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c004d40 0x7f242c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f2449b6a640 1 -- 172.21.15.28:0/2507684662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242c003520 con 0x7f242c0039b0 2024-04-25T19:12:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.345+0000 7f244a36b640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f242c004d40 0x7f242c005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.346+0000 7f2449b6a640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f242c003e20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f243c009ac0 tx=0x7f242c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca2bedf0eb787a88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.346+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c036070 con 0x7f242c0039b0 2024-04-25T19:12:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.346+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c031400 con 0x7f242c0039b0 2024-04-25T19:12:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.346+0000 7f2448b68640 1 -- 172.21.15.28:0/2507684662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c009690 con 0x7f242c0039b0 2024-04-25T19:12:11.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.347+0000 7f244bdce640 1 -- 172.21.15.28:0/2507684662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 msgr2=0x7f242c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.347+0000 7f244bdce640 1 --2- 172.21.15.28:0/2507684662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f242c003e20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f243c009ac0 tx=0x7f242c005830 comp rx=0 tx=0).stop 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.347+0000 7f244a36b640 1 -- 172.21.15.28:0/2507684662 reap_dead start 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.347+0000 7f244bdce640 1 -- 172.21.15.28:0/2507684662 shutdown_connections 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.347+0000 7f244bdce640 1 -- 172.21.15.28:0/2507684662 >> 172.21.15.28:0/2507684662 conn(0x7f24440b7640 msgr2=0x7f24440b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.347+0000 7f244bdce640 1 -- 172.21.15.28:0/2507684662 shutdown_connections 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.347+0000 7f244bdce640 1 -- 172.21.15.28:0/2507684662 wait complete. 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.349+0000 7f244bdce640 1 Processor -- start 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.349+0000 7f244bdce640 1 -- start start 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f244bdce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f24441fa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f244bdce640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2444150750 0x7f244415dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f244bdce640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f244415e2c0 0x7f244409cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f244bdce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24440cf290 con 0x7f242c0039b0 2024-04-25T19:12:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f244bdce640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f24440cf110 con 0x7f244415e2c0 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f244bdce640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f24441614d0 con 0x7f2444150750 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f2449b6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f24441fa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f2449b6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f24441fa8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56078/0 (socket says 172.21.15.28:56078) 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f2449b6a640 1 -- 172.21.15.28:0/2892525087 learned_addr learned my addr 172.21.15.28:0/2892525087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f244a36b640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f244415e2c0 0x7f244409cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f2449369640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2444150750 0x7f244415dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f2449b6a640 1 -- 172.21.15.28:0/2892525087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2444150750 msgr2=0x7f244415dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f2449b6a640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2444150750 0x7f244415dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f2449b6a640 1 -- 172.21.15.28:0/2892525087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f244415e2c0 msgr2=0x7f244409cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.350+0000 7f2449b6a640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f244415e2c0 0x7f244409cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.351+0000 7f2449b6a640 1 -- 172.21.15.28:0/2892525087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2444150e90 con 0x7f242c0039b0 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.351+0000 7f2449b6a640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f24441fa8c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f243c009be0 tx=0x7f243c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.351+0000 7f24427fc640 1 -- 172.21.15.28:0/2892525087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c040020 con 0x7f242c0039b0 2024-04-25T19:12:11.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.351+0000 7f24427fc640 1 -- 172.21.15.28:0/2892525087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f243c003b40 con 0x7f242c0039b0 2024-04-25T19:12:11.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.351+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2444151170 con 0x7f242c0039b0 2024-04-25T19:12:11.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.351+0000 7f24427fc640 1 -- 172.21.15.28:0/2892525087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f243c036040 con 0x7f242c0039b0 2024-04-25T19:12:11.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.351+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f244409c8c0 con 0x7f242c0039b0 2024-04-25T19:12:11.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.352+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24440d1290 con 0x7f242c0039b0 2024-04-25T19:12:11.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.353+0000 7f24427fc640 1 -- 172.21.15.28:0/2892525087 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f243c043020 con 0x7f242c0039b0 2024-04-25T19:12:11.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.353+0000 7f24427fc640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24300744b0 0x7f24300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.353+0000 7f24427fc640 1 -- 172.21.15.28:0/2892525087 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f243c0b5930 con 0x7f242c0039b0 2024-04-25T19:12:11.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.353+0000 7f2449369640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24300744b0 0x7f24300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.354+0000 7f2449369640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24300744b0 0x7f24300769c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f24440cea70 tx=0x7f2438009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.358+0000 7f24427fc640 1 -- 172.21.15.28:0/2892525087 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f24440d1290 con 0x7f242c0039b0 2024-04-25T19:12:11.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.542+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2444050740 con 0x7f242c0039b0 2024-04-25T19:12:11.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.542+0000 7f24427fc640 1 -- 172.21.15.28:0/2892525087 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f243c04a030 con 0x7f242c0039b0 2024-04-25T19:12:11.544 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24300744b0 msgr2=0x7f24300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24300744b0 0x7f24300769c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f24440cea70 tx=0x7f2438009190 comp rx=0 tx=0).stop 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 msgr2=0x7f24441fa8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f24441fa8c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f243c009be0 tx=0x7f243c004730 comp rx=0 tx=0).stop 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 shutdown_connections 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24300744b0 0x7f24300769c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f244415e2c0 0x7f244409cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2444150750 0x7f244415dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 --2- 172.21.15.28:0/2892525087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f242c0039b0 0x7f24441fa8c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 >> 172.21.15.28:0/2892525087 conn(0x7f24440b7640 msgr2=0x7f244406e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 shutdown_connections 2024-04-25T19:12:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.544+0000 7f244bdce640 1 -- 172.21.15.28:0/2892525087 wait complete. 2024-04-25T19:12:11.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.667+0000 7f0c345a8640 1 Processor -- start 2024-04-25T19:12:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.668+0000 7f0c345a8640 1 -- start start 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.669+0000 7f0c345a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.669+0000 7f0c345a8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c2c0d1ce0 con 0x7f0c2c0b7710 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.669+0000 7f0c345a8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c2c0cf8d0 con 0x7f0c2c09caf0 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.669+0000 7f0c345a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2c0b8fa0 con 0x7f0c2c09cf00 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.669+0000 7f0c32b45640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0c2c0b7710 0x7f0c2c0ce720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50496/0 (socket says 172.21.15.28:50496) 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.669+0000 7f0c32b45640 1 -- 172.21.15.28:0/2781882703 learned_addr learned my addr 172.21.15.28:0/2781882703 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.669+0000 7f0c31b43640 1 --2- 172.21.15.28:0/2781882703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.670+0000 7f0c31342640 1 -- 172.21.15.28:0/2781882703 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2017219675 0 0) 0x7f0c2c0d1ce0 con 0x7f0c2c0b7710 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.670+0000 7f0c31342640 1 -- 172.21.15.28:0/2781882703 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c14003520 con 0x7f0c2c0b7710 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.670+0000 7f0c31342640 1 -- 172.21.15.28:0/2781882703 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 260158101 0 0) 0x7f0c2c0cf8d0 con 0x7f0c2c09caf0 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.670+0000 7f0c31342640 1 -- 172.21.15.28:0/2781882703 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c2c0d1ce0 con 0x7f0c2c09caf0 2024-04-25T19:12:11.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.670+0000 7f0c31b43640 1 -- 172.21.15.28:0/2781882703 >> v1:172.21.15.53:6789/0 conn(0x7f0c2c09caf0 legacy=0x7f0c2c0d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.670+0000 7f0c31b43640 1 -- 172.21.15.28:0/2781882703 >> v1:172.21.15.28:6789/0 conn(0x7f0c2c0b7710 legacy=0x7f0c2c0ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.670+0000 7f0c31b43640 1 -- 172.21.15.28:0/2781882703 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2c0cfbb0 con 0x7f0c2c09cf00 2024-04-25T19:12:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.671+0000 7f0c31b43640 1 --2- 172.21.15.28:0/2781882703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b71d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0c200097d0 tx=0x7f0c2002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=260e0b1c2fc0be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.671+0000 7f0c31342640 1 -- 172.21.15.28:0/2781882703 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c20037070 con 0x7f0c2c09cf00 2024-04-25T19:12:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.671+0000 7f0c31342640 1 -- 172.21.15.28:0/2781882703 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c20032440 con 0x7f0c2c09cf00 2024-04-25T19:12:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.671+0000 7f0c31342640 1 -- 172.21.15.28:0/2781882703 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c200327f0 con 0x7f0c2c09cf00 2024-04-25T19:12:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.672+0000 7f0c345a8640 1 -- 172.21.15.28:0/2781882703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 msgr2=0x7f0c2c0b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.672+0000 7f0c345a8640 1 --2- 172.21.15.28:0/2781882703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b71d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0c200097d0 tx=0x7f0c2002a180 comp rx=0 tx=0).stop 2024-04-25T19:12:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.672+0000 7f0c345a8640 1 -- 172.21.15.28:0/2781882703 shutdown_connections 2024-04-25T19:12:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.672+0000 7f0c345a8640 1 --2- 172.21.15.28:0/2781882703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b71d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.672+0000 7f0c345a8640 1 -- 172.21.15.28:0/2781882703 >> 172.21.15.28:0/2781882703 conn(0x7f0c2c0d03f0 msgr2=0x7f0c2c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.672+0000 7f0c345a8640 1 -- 172.21.15.28:0/2781882703 shutdown_connections 2024-04-25T19:12:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.672+0000 7f0c345a8640 1 -- 172.21.15.28:0/2781882703 wait complete. 2024-04-25T19:12:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.673+0000 7f0c345a8640 1 Processor -- start 2024-04-25T19:12:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.674+0000 7f0c345a8640 1 -- start start 2024-04-25T19:12:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c345a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c09caf0 0x7f0c2c1dbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c345a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c345a8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2c0b7710 0x7f0c2c0b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c345a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c0b9430 con 0x7f0c2c09caf0 2024-04-25T19:12:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c345a8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2c0d19e0 con 0x7f0c2c0b7710 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c345a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0c2c0d4790 con 0x7f0c2c09cf00 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c09caf0 0x7f0c2c1dbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c09caf0 0x7f0c2c1dbb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56088/0 (socket says 172.21.15.28:56088) 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 -- 172.21.15.28:0/1283968200 learned_addr learned my addr 172.21.15.28:0/1283968200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32b45640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2c0b7710 0x7f0c2c0b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c31b43640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 -- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 msgr2=0x7f0c2c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 -- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2c0b7710 msgr2=0x7f0c2c0b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2c0b7710 0x7f0c2c0b10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 -- 172.21.15.28:0/1283968200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c2c0b1930 con 0x7f0c2c09caf0 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32b45640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2c0b7710 0x7f0c2c0b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.675+0000 7f0c32344640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c09caf0 0x7f0c2c1dbb50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400c3f0 tx=0x7f0c2400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.676+0000 7f0c2affd640 1 -- 172.21.15.28:0/1283968200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c24017070 con 0x7f0c2c09caf0 2024-04-25T19:12:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.676+0000 7f0c2affd640 1 -- 172.21.15.28:0/1283968200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c240052f0 con 0x7f0c2c09caf0 2024-04-25T19:12:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.676+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c2c0b1c10 con 0x7f0c2c09caf0 2024-04-25T19:12:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.676+0000 7f0c2affd640 1 -- 172.21.15.28:0/1283968200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c24003830 con 0x7f0c2c09caf0 2024-04-25T19:12:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.676+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c2c0b21d0 con 0x7f0c2c09caf0 2024-04-25T19:12:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.677+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c2c050740 con 0x7f0c2c09caf0 2024-04-25T19:12:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.677+0000 7f0c2affd640 1 -- 172.21.15.28:0/1283968200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0c2401a070 con 0x7f0c2c09caf0 2024-04-25T19:12:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.678+0000 7f0c2affd640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c180744b0 0x7f0c180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.678+0000 7f0c2affd640 1 -- 172.21.15.28:0/1283968200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0c240949e0 con 0x7f0c2c09caf0 2024-04-25T19:12:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.678+0000 7f0c31b43640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c180744b0 0x7f0c180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.678+0000 7f0c31b43640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c180744b0 0x7f0c180769c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0c20034000 tx=0x7f0c2002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.683+0000 7f0c2affd640 1 -- 172.21.15.28:0/1283968200 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c2405ed70 con 0x7f0c2c09caf0 2024-04-25T19:12:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.868+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c2c071620 con 0x7f0c2c09caf0 2024-04-25T19:12:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.869+0000 7f0c2affd640 1 -- 172.21.15.28:0/1283968200 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f0c2405e670 con 0x7f0c2c09caf0 2024-04-25T19:12:11.871 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:11.871 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:11.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c180744b0 msgr2=0x7f0c180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c180744b0 0x7f0c180769c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0c20034000 tx=0x7f0c2002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:11.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c09caf0 msgr2=0x7f0c2c1dbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c09caf0 0x7f0c2c1dbb50 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400c3f0 tx=0x7f0c2400c8b0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 shutdown_connections 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0c180744b0 0x7f0c180769c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c2c0b7710 0x7f0c2c0b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0c2c09cf00 0x7f0c2c0b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 --2- 172.21.15.28:0/1283968200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2c09caf0 0x7f0c2c1dbb50 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 >> 172.21.15.28:0/1283968200 conn(0x7f0c2c0d03f0 msgr2=0x7f0c2c071ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 shutdown_connections 2024-04-25T19:12:11.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.871+0000 7f0c345a8640 1 -- 172.21.15.28:0/1283968200 wait complete. 2024-04-25T19:12:11.885 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:11.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.999+0000 7f5ca498f640 1 Processor -- start 2024-04-25T19:12:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:11.999+0000 7f5ca498f640 1 -- start start 2024-04-25T19:12:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.000+0000 7f5ca498f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca00b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.000+0000 7f5ca498f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ca0164b20 con 0x7f5ca00d2ca0 2024-04-25T19:12:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.000+0000 7f5ca498f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ca0163d60 con 0x7f5ca00b81a0 2024-04-25T19:12:12.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.000+0000 7f5ca498f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5ca00b4f50 con 0x7f5ca0160250 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9ed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ca00d2ca0 0x7f5ca00d44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50502/0 (socket says 172.21.15.28:50502) 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9ed9d640 1 -- 172.21.15.28:0/3301839942 learned_addr learned my addr 172.21.15.28:0/3301839942 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9e59c640 1 --2- 172.21.15.28:0/3301839942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca00b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9d59a640 1 -- 172.21.15.28:0/3301839942 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1289873306 0 0) 0x7f5ca0164b20 con 0x7f5ca00d2ca0 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9d59a640 1 -- 172.21.15.28:0/3301839942 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c8c003520 con 0x7f5ca00d2ca0 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9d59a640 1 -- 172.21.15.28:0/3301839942 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2801212739 0 0) 0x7f5ca0163d60 con 0x7f5ca00b81a0 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9d59a640 1 -- 172.21.15.28:0/3301839942 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ca0164b20 con 0x7f5ca00b81a0 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9e59c640 1 -- 172.21.15.28:0/3301839942 >> v1:172.21.15.53:6789/0 conn(0x7f5ca00b81a0 legacy=0x7f5ca00d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9e59c640 1 -- 172.21.15.28:0/3301839942 >> v1:172.21.15.28:6789/0 conn(0x7f5ca00d2ca0 legacy=0x7f5ca00d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.001+0000 7f5c9e59c640 1 -- 172.21.15.28:0/3301839942 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca0164040 con 0x7f5ca0160250 2024-04-25T19:12:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.002+0000 7f5c9e59c640 1 --2- 172.21.15.28:0/3301839942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca00b4690 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c880097d0 tx=0x7f5c8802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1823a10c35b64d87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.002+0000 7f5c9d59a640 1 -- 172.21.15.28:0/3301839942 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c88037070 con 0x7f5ca0160250 2024-04-25T19:12:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.002+0000 7f5c9d59a640 1 -- 172.21.15.28:0/3301839942 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c88032440 con 0x7f5ca0160250 2024-04-25T19:12:12.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.002+0000 7f5c9d59a640 1 -- 172.21.15.28:0/3301839942 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c880327f0 con 0x7f5ca0160250 2024-04-25T19:12:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.003+0000 7f5ca498f640 1 -- 172.21.15.28:0/3301839942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 msgr2=0x7f5ca00b4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.003+0000 7f5ca498f640 1 --2- 172.21.15.28:0/3301839942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca00b4690 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5c880097d0 tx=0x7f5c8802a180 comp rx=0 tx=0).stop 2024-04-25T19:12:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.003+0000 7f5ca498f640 1 -- 172.21.15.28:0/3301839942 shutdown_connections 2024-04-25T19:12:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.003+0000 7f5ca498f640 1 --2- 172.21.15.28:0/3301839942 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca00b4690 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.003+0000 7f5ca498f640 1 -- 172.21.15.28:0/3301839942 >> 172.21.15.28:0/3301839942 conn(0x7f5ca006ee30 msgr2=0x7f5ca006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.003+0000 7f5ca498f640 1 -- 172.21.15.28:0/3301839942 shutdown_connections 2024-04-25T19:12:12.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.003+0000 7f5ca498f640 1 -- 172.21.15.28:0/3301839942 wait complete. 2024-04-25T19:12:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.005+0000 7f5ca498f640 1 Processor -- start 2024-04-25T19:12:12.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.005+0000 7f5ca498f640 1 -- start start 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5ca498f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00b81a0 0x7f5ca0152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5ca498f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca01f91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5ca498f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca0154250 0x7f5ca015eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5ca498f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca00b4bd0 con 0x7f5ca0154250 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5ca498f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ca0164600 con 0x7f5ca00b81a0 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5ca498f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5ca00d4bc0 con 0x7f5ca0160250 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5c9ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca0154250 0x7f5ca015eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5c9ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca0154250 0x7f5ca015eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56094/0 (socket says 172.21.15.28:56094) 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.006+0000 7f5c9ed9d640 1 -- 172.21.15.28:0/3650999614 learned_addr learned my addr 172.21.15.28:0/3650999614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:12.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9dd9b640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca01f91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9e59c640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00b81a0 0x7f5ca0152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9ed9d640 1 -- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 msgr2=0x7f5ca01f91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9ed9d640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca01f91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9ed9d640 1 -- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00b81a0 msgr2=0x7f5ca0152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9ed9d640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00b81a0 0x7f5ca0152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9ed9d640 1 -- 172.21.15.28:0/3650999614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ca01f9710 con 0x7f5ca0154250 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9dd9b640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca01f91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9e59c640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00b81a0 0x7f5ca0152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c9ed9d640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca0154250 0x7f5ca015eec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800cb10 tx=0x7f5c9800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c96ffd640 1 -- 172.21.15.28:0/3650999614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98017070 con 0x7f5ca0154250 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c96ffd640 1 -- 172.21.15.28:0/3650999614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c9800ac70 con 0x7f5ca0154250 2024-04-25T19:12:12.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ca0154a30 con 0x7f5ca0154250 2024-04-25T19:12:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5c96ffd640 1 -- 172.21.15.28:0/3650999614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c98012600 con 0x7f5ca0154250 2024-04-25T19:12:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.007+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ca0157690 con 0x7f5ca0154250 2024-04-25T19:12:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.008+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ca01547a0 con 0x7f5ca0154250 2024-04-25T19:12:12.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.009+0000 7f5c96ffd640 1 -- 172.21.15.28:0/3650999614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5c9801a070 con 0x7f5ca0154250 2024-04-25T19:12:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.009+0000 7f5c96ffd640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c800744b0 0x7f5c800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.009+0000 7f5c96ffd640 1 -- 172.21.15.28:0/3650999614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5c98095290 con 0x7f5ca0154250 2024-04-25T19:12:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.009+0000 7f5c9e59c640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c800744b0 0x7f5c800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.010+0000 7f5c9e59c640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c800744b0 0x7f5c800769c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5c88034000 tx=0x7f5c8802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.014+0000 7f5c96ffd640 1 -- 172.21.15.28:0/3650999614 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c9800ae20 con 0x7f5ca0154250 2024-04-25T19:12:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.203+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5ca0050740 con 0x7f5ca0154250 2024-04-25T19:12:12.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.204+0000 7f5c96ffd640 1 -- 172.21.15.28:0/3650999614 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f5c9805efd0 con 0x7f5ca0154250 2024-04-25T19:12:12.206 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c800744b0 msgr2=0x7f5c800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c800744b0 0x7f5c800769c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5c88034000 tx=0x7f5c8802e000 comp rx=0 tx=0).stop 2024-04-25T19:12:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca0154250 msgr2=0x7f5ca015eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca0154250 0x7f5ca015eec0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5c9800cb10 tx=0x7f5c9800a060 comp rx=0 tx=0).stop 2024-04-25T19:12:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 shutdown_connections 2024-04-25T19:12:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c800744b0 0x7f5c800769c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ca0154250 0x7f5ca015eec0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ca0160250 0x7f5ca01f91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 --2- 172.21.15.28:0/3650999614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ca00b81a0 0x7f5ca0152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 >> 172.21.15.28:0/3650999614 conn(0x7f5ca006ee30 msgr2=0x7f5ca00b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 shutdown_connections 2024-04-25T19:12:12.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.206+0000 7f5ca498f640 1 -- 172.21.15.28:0/3650999614 wait complete. 2024-04-25T19:12:12.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.331+0000 7f81dbe4c640 1 Processor -- start 2024-04-25T19:12:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.331+0000 7f81dbe4c640 1 -- start start 2024-04-25T19:12:12.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81dbe4c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40d2ca0 0x7f81d416afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81dbe4c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d4164b20 con 0x7f81d4160250 2024-04-25T19:12:12.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81dbe4c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d4163d60 con 0x7f81d40b81a0 2024-04-25T19:12:12.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81dbe4c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f81d40b4f50 con 0x7f81d40d2ca0 2024-04-25T19:12:12.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81d9be8640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f81d4160250 0x7f81d40b4690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50506/0 (socket says 172.21.15.28:50506) 2024-04-25T19:12:12.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81d9be8640 1 -- 172.21.15.28:0/1466114784 learned_addr learned my addr 172.21.15.28:0/1466114784 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81da3e9640 1 --2- 172.21.15.28:0/1466114784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40d2ca0 0x7f81d416afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81d8be6640 1 -- 172.21.15.28:0/1466114784 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3013489181 0 0) 0x7f81d4164b20 con 0x7f81d4160250 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81d8be6640 1 -- 172.21.15.28:0/1466114784 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81bc003520 con 0x7f81d4160250 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81d8be6640 1 -- 172.21.15.28:0/1466114784 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323398312 0 0) 0x7f81d4163d60 con 0x7f81d40b81a0 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81d8be6640 1 -- 172.21.15.28:0/1466114784 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81d4164b20 con 0x7f81d40b81a0 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81da3e9640 1 -- 172.21.15.28:0/1466114784 >> v1:172.21.15.53:6789/0 conn(0x7f81d40b81a0 legacy=0x7f81d40d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81da3e9640 1 -- 172.21.15.28:0/1466114784 >> v1:172.21.15.28:6789/0 conn(0x7f81d4160250 legacy=0x7f81d40b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.333+0000 7f81d8be6640 1 -- 172.21.15.28:0/1466114784 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 125954555 0 0) 0x7f81bc003520 con 0x7f81d4160250 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.334+0000 7f81da3e9640 1 -- 172.21.15.28:0/1466114784 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d4164040 con 0x7f81d40d2ca0 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.334+0000 7f81da3e9640 1 --2- 172.21.15.28:0/1466114784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40d2ca0 0x7f81d416afc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009dc0 tx=0x7f81c402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c42d4a1fe210d87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81d8be6640 1 -- 172.21.15.28:0/1466114784 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc001690 con 0x7f81d40d2ca0 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81d8be6640 1 -- 172.21.15.28:0/1466114784 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c4007540 con 0x7f81d40d2ca0 2024-04-25T19:12:12.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81d8be6640 1 -- 172.21.15.28:0/1466114784 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c40078f0 con 0x7f81d40d2ca0 2024-04-25T19:12:12.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81dbe4c640 1 -- 172.21.15.28:0/1466114784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40d2ca0 msgr2=0x7f81d416afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81dbe4c640 1 --2- 172.21.15.28:0/1466114784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40d2ca0 0x7f81d416afc0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009dc0 tx=0x7f81c402a5d0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81dbe4c640 1 -- 172.21.15.28:0/1466114784 shutdown_connections 2024-04-25T19:12:12.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81dbe4c640 1 --2- 172.21.15.28:0/1466114784 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40d2ca0 0x7f81d416afc0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81dbe4c640 1 -- 172.21.15.28:0/1466114784 >> 172.21.15.28:0/1466114784 conn(0x7f81d406ee30 msgr2=0x7f81d406c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:12.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81dbe4c640 1 -- 172.21.15.28:0/1466114784 shutdown_connections 2024-04-25T19:12:12.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.335+0000 7f81dbe4c640 1 -- 172.21.15.28:0/1466114784 wait complete. 2024-04-25T19:12:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.337+0000 7f81dbe4c640 1 Processor -- start 2024-04-25T19:12:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.337+0000 7f81dbe4c640 1 -- start start 2024-04-25T19:12:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.338+0000 7f81dbe4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d40b81a0 0x7f81d4161480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.338+0000 7f81dbe4c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81d4160250 0x7f81d4161cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.338+0000 7f81dbe4c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40bb4b0 0x7f81d4152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.338+0000 7f81dbe4c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f81d40d54c0 con 0x7f81d40b81a0 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.338+0000 7f81dbe4c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f81d4164600 con 0x7f81d4160250 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.338+0000 7f81dbe4c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f81d416b500 con 0x7f81d40bb4b0 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d40b81a0 0x7f81d4161480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d40b81a0 0x7f81d4161480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56098/0 (socket says 172.21.15.28:56098) 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 -- 172.21.15.28:0/393350982 learned_addr learned my addr 172.21.15.28:0/393350982 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81da3e9640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40bb4b0 0x7f81d4152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d93e7640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81d4160250 0x7f81d4161cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 -- 172.21.15.28:0/393350982 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40bb4b0 msgr2=0x7f81d4152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40bb4b0 0x7f81d4152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 -- 172.21.15.28:0/393350982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81d4160250 msgr2=0x7f81d4161cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81d4160250 0x7f81d4161cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 -- 172.21.15.28:0/393350982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d4160700 con 0x7f81d40b81a0 2024-04-25T19:12:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d9be8640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d40b81a0 0x7f81d4161480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00c4b0 tx=0x7f81cc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d27fc640 1 -- 172.21.15.28:0/393350982 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc017070 con 0x7f81d40b81a0 2024-04-25T19:12:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d27fc640 1 -- 172.21.15.28:0/393350982 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81cc005ce0 con 0x7f81d40b81a0 2024-04-25T19:12:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81d27fc640 1 -- 172.21.15.28:0/393350982 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81cc007290 con 0x7f81d40b81a0 2024-04-25T19:12:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d4162210 con 0x7f81d40b81a0 2024-04-25T19:12:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.339+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d4160c80 con 0x7f81d40b81a0 2024-04-25T19:12:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.340+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d4050740 con 0x7f81d40b81a0 2024-04-25T19:12:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.341+0000 7f81d27fc640 1 -- 172.21.15.28:0/393350982 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f81cc01a070 con 0x7f81d40b81a0 2024-04-25T19:12:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.341+0000 7f81d27fc640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c00744b0 0x7f81c00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.342+0000 7f81d93e7640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c00744b0 0x7f81c00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.342+0000 7f81d27fc640 1 -- 172.21.15.28:0/393350982 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f81cc0949e0 con 0x7f81d40b81a0 2024-04-25T19:12:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.342+0000 7f81d93e7640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c00744b0 0x7f81c00769c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f81d41583f0 tx=0x7f81c8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.346+0000 7f81d27fc640 1 -- 172.21.15.28:0/393350982 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81cc05e680 con 0x7f81d40b81a0 2024-04-25T19:12:12.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.532+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f81d4155a20 con 0x7f81d40b81a0 2024-04-25T19:12:12.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.533+0000 7f81d27fc640 1 -- 172.21.15.28:0/393350982 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f81cc062980 con 0x7f81d40b81a0 2024-04-25T19:12:12.534 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:12.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c00744b0 msgr2=0x7f81c00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c00744b0 0x7f81c00769c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f81d41583f0 tx=0x7f81c8009040 comp rx=0 tx=0).stop 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d40b81a0 msgr2=0x7f81d4161480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d40b81a0 0x7f81d4161480 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f81cc00c4b0 tx=0x7f81cc00c970 comp rx=0 tx=0).stop 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 shutdown_connections 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f81c00744b0 0x7f81c00769c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f81d40bb4b0 0x7f81d4152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f81d4160250 0x7f81d4161cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 --2- 172.21.15.28:0/393350982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81d40b81a0 0x7f81d4161480 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.534+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 >> 172.21.15.28:0/393350982 conn(0x7f81d406ee30 msgr2=0x7f81d4158b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.535+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 shutdown_connections 2024-04-25T19:12:12.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.535+0000 7f81dbe4c640 1 -- 172.21.15.28:0/393350982 wait complete. 2024-04-25T19:12:12.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:12.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.662+0000 7f9ea159d640 1 Processor -- start 2024-04-25T19:12:12.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.662+0000 7f9ea159d640 1 -- start start 2024-04-25T19:12:12.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.663+0000 7f9ea159d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c0b7af0 0x7f9e9c15fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.663+0000 7f9ea159d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9c0d13c0 con 0x7f9e9c0d0fb0 2024-04-25T19:12:12.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.663+0000 7f9ea159d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e9c163f10 con 0x7f9e9c0b8e50 2024-04-25T19:12:12.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.663+0000 7f9ea159d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9c1641f0 con 0x7f9e9c0b7af0 2024-04-25T19:12:12.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9bfff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9e9c0d0fb0 0x7f9e9c0b73e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50522/0 (socket says 172.21.15.28:50522) 2024-04-25T19:12:12.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9bfff640 1 -- 172.21.15.28:0/751021361 learned_addr learned my addr 172.21.15.28:0/751021361 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:12.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9b7fe640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c0b7af0 0x7f9e9c15fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3723247053 0 0) 0x7f9e9c0d13c0 con 0x7f9e9c0d0fb0 2024-04-25T19:12:12.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e88003520 con 0x7f9e9c0d0fb0 2024-04-25T19:12:12.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736719563 0 0) 0x7f9e9c163f10 con 0x7f9e9c0b8e50 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e9c0d13c0 con 0x7f9e9c0b8e50 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 891979670 0 0) 0x7f9e88003520 con 0x7f9e9c0d0fb0 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c0b7af0 msgr2=0x7f9e9c15fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c0b7af0 0x7f9e9c15fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 >> v1:172.21.15.53:6789/0 conn(0x7f9e9c0b8e50 legacy=0x7f9e9c0d45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c1644d0 con 0x7f9e9c0d0fb0 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f9e84001670 con 0x7f9e9c0d0fb0 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 >> v1:172.21.15.28:6789/0 conn(0x7f9e9c0d0fb0 legacy=0x7f9e9c0b73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e88003b70 0x7f9e88003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e88004800 0x7f9e88004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e88005150 0x7f9e8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c1641f0 con 0x7f9e88005150 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e8800c030 con 0x7f9e88004800 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9e8800c230 con 0x7f9e88003b70 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9ea0d9c640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e88005150 0x7f9e8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9b7fe640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e88004800 0x7f9e88004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.664+0000 7f9e9bfff640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e88003b70 0x7f9e88003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.665+0000 7f9ea0d9c640 1 -- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e88003b70 msgr2=0x7f9e88003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.665+0000 7f9ea0d9c640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e88003b70 0x7f9e88003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.665+0000 7f9ea0d9c640 1 -- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e88004800 msgr2=0x7f9e88004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.665+0000 7f9ea0d9c640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e88004800 0x7f9e88004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.665+0000 7f9ea0d9c640 1 -- 172.21.15.28:0/751021361 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e88003520 con 0x7f9e88005150 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9ea0d9c640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e88005150 0x7f9e8800b9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c164ad0 tx=0x7f9e88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a441de432fd509e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94001bf0 con 0x7f9e88005150 2024-04-25T19:12:12.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e94003b70 con 0x7f9e88005150 2024-04-25T19:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9e9affd640 1 -- 172.21.15.28:0/751021361 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e940050d0 con 0x7f9e88005150 2024-04-25T19:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9ea159d640 1 -- 172.21.15.28:0/751021361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e88005150 msgr2=0x7f9e8800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9ea159d640 1 --2- 172.21.15.28:0/751021361 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e88005150 0x7f9e8800b9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c164ad0 tx=0x7f9e88005830 comp rx=0 tx=0).stop 2024-04-25T19:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9ea0d9c640 1 -- 172.21.15.28:0/751021361 reap_dead start 2024-04-25T19:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9ea159d640 1 -- 172.21.15.28:0/751021361 shutdown_connections 2024-04-25T19:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9ea159d640 1 -- 172.21.15.28:0/751021361 >> 172.21.15.28:0/751021361 conn(0x7f9e9c0d23e0 msgr2=0x7f9e9c0d27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9ea159d640 1 -- 172.21.15.28:0/751021361 shutdown_connections 2024-04-25T19:12:12.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.666+0000 7f9ea159d640 1 -- 172.21.15.28:0/751021361 wait complete. 2024-04-25T19:12:12.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.668+0000 7f9ea159d640 1 Processor -- start 2024-04-25T19:12:12.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.668+0000 7f9ea159d640 1 -- start start 2024-04-25T19:12:12.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9ea159d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c1618f0 0x7f9e9c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9ea159d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e9c15a300 0x7f9e9c1f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9ea159d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9c15c0a0 0x7f9e9c15c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9ea159d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c0b9260 con 0x7f9e9c15a300 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9ea159d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c0b4bc0 con 0x7f9e9c15c0a0 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9ea159d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9c163a80 con 0x7f9e9c1618f0 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e9c15a300 0x7f9e9c1f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e9c15a300 0x7f9e9c1f9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56116/0 (socket says 172.21.15.28:56116) 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 -- 172.21.15.28:0/1975954322 learned_addr learned my addr 172.21.15.28:0/1975954322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9bfff640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c1618f0 0x7f9e9c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9ea0d9c640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9c15c0a0 0x7f9e9c15c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 -- 172.21.15.28:0/1975954322 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c1618f0 msgr2=0x7f9e9c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c1618f0 0x7f9e9c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 -- 172.21.15.28:0/1975954322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9c15c0a0 msgr2=0x7f9e9c15c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9c15c0a0 0x7f9e9c15c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 -- 172.21.15.28:0/1975954322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c15c9f0 con 0x7f9e9c15a300 2024-04-25T19:12:12.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e9b7fe640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e9c15a300 0x7f9e9c1f9ad0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0119d0 tx=0x7f9e8c011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e98ff9640 1 -- 172.21.15.28:0/1975954322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00fb50 con 0x7f9e9c15a300 2024-04-25T19:12:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e98ff9640 1 -- 172.21.15.28:0/1975954322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e8c00fcf0 con 0x7f9e9c15a300 2024-04-25T19:12:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9c1f9310 con 0x7f9e9c15a300 2024-04-25T19:12:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.670+0000 7f9e98ff9640 1 -- 172.21.15.28:0/1975954322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8c00fea0 con 0x7f9e9c15a300 2024-04-25T19:12:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.671+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9c1622d0 con 0x7f9e9c15a300 2024-04-25T19:12:12.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.672+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e9c050740 con 0x7f9e9c15a300 2024-04-25T19:12:12.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.672+0000 7f9e98ff9640 1 -- 172.21.15.28:0/1975954322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9e8c012040 con 0x7f9e9c15a300 2024-04-25T19:12:12.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.673+0000 7f9e98ff9640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9e7c0744b0 0x7f9e7c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.673+0000 7f9e98ff9640 1 -- 172.21.15.28:0/1975954322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9e8c095b60 con 0x7f9e9c15a300 2024-04-25T19:12:12.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.673+0000 7f9e9bfff640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9e7c0744b0 0x7f9e7c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.673+0000 7f9e9bfff640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9e7c0744b0 0x7f9e7c0769c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c161880 tx=0x7f9e84000fc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:12.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.678+0000 7f9e98ff9640 1 -- 172.21.15.28:0/1975954322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9e9c15c9f0 con 0x7f9e9c15a300 2024-04-25T19:12:12.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.863+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9e9c162510 con 0x7f9e9c15a300 2024-04-25T19:12:12.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.863+0000 7f9e98ff9640 1 -- 172.21.15.28:0/1975954322 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f9e8c019030 con 0x7f9e9c15a300 2024-04-25T19:12:12.865 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:12.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9e7c0744b0 msgr2=0x7f9e7c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9e7c0744b0 0x7f9e7c0769c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c161880 tx=0x7f9e84000fc0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e9c15a300 msgr2=0x7f9e9c1f9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e9c15a300 0x7f9e9c1f9ad0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f9e8c0119d0 tx=0x7f9e8c011e90 comp rx=0 tx=0).stop 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 shutdown_connections 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9e7c0744b0 0x7f9e7c0769c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e9c15c0a0 0x7f9e9c15c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e9c15a300 0x7f9e9c1f9ad0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 --2- 172.21.15.28:0/1975954322 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9e9c1618f0 0x7f9e9c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.865+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 >> 172.21.15.28:0/1975954322 conn(0x7f9e9c0d23e0 msgr2=0x7f9e9c06df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.866+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 shutdown_connections 2024-04-25T19:12:12.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.866+0000 7f9ea159d640 1 -- 172.21.15.28:0/1975954322 wait complete. 2024-04-25T19:12:12.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.992+0000 7efd6ea34640 1 Processor -- start 2024-04-25T19:12:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.992+0000 7efd6ea34640 1 -- start start 2024-04-25T19:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.993+0000 7efd6ea34640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68160310 0x7efd680d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.993+0000 7efd6ea34640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efd680b9920 0x7efd680b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.993+0000 7efd6ea34640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd68160720 con 0x7efd680b79f0 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.993+0000 7efd6ea34640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7efd680d5640 con 0x7efd680b9920 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.993+0000 7efd6ea34640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efd680d0f40 con 0x7efd68160310 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd677fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efd680b79f0 0x7efd680b7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50524/0 (socket says 172.21.15.28:50524) 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd677fe640 1 -- 172.21.15.28:0/147002675 learned_addr learned my addr 172.21.15.28:0/147002675 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2860384705 0 0) 0x7efd68160720 con 0x7efd680b79f0 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd6cfd1640 1 --2- 172.21.15.28:0/147002675 >> v2:172.21.15.53:3300/0 conn(0x7efd680b9920 0x7efd680b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd54003520 con 0x7efd680b79f0 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd67fff640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68160310 0x7efd680d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1753891862 0 0) 0x7efd54003520 con 0x7efd680b79f0 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68160310 msgr2=0x7efd680d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68160310 0x7efd680d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:12.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 >> v2:172.21.15.53:3300/0 conn(0x7efd680b9920 msgr2=0x7efd680b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 --2- 172.21.15.28:0/147002675 >> v2:172.21.15.53:3300/0 conn(0x7efd680b9920 0x7efd680b9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd680d1220 con 0x7efd680b79f0 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7efd58001670 con 0x7efd680b79f0 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 >> v1:172.21.15.28:6789/0 conn(0x7efd680b79f0 legacy=0x7efd680b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd540039b0 0x7efd54003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68160310 0x7efd54004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd54004d40 0x7efd54005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd680d5640 con 0x7efd54004d40 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd680d0f40 con 0x7efd540039b0 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efd5400b9c0 con 0x7efd68160310 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd6cfd1640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd54004d40 0x7efd54005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd67fff640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd540039b0 0x7efd54003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.994+0000 7efd677fe640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68160310 0x7efd54004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.995+0000 7efd6cfd1640 1 -- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68160310 msgr2=0x7efd54004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.995+0000 7efd6cfd1640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68160310 0x7efd54004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.995+0000 7efd6cfd1640 1 -- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd540039b0 msgr2=0x7efd54003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.995+0000 7efd6cfd1640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd540039b0 0x7efd54003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.995+0000 7efd6cfd1640 1 -- 172.21.15.28:0/147002675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd54003520 con 0x7efd54004d40 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.995+0000 7efd6cfd1640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd54004d40 0x7efd54005150 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd60009f90 tx=0x7efd54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f7adb90f7e01373 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60009c10 con 0x7efd54004d40 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60036070 con 0x7efd54004d40 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd66ffd640 1 -- 172.21.15.28:0/147002675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60031880 con 0x7efd54004d40 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd6ea34640 1 -- 172.21.15.28:0/147002675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd54004d40 msgr2=0x7efd54005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd6ea34640 1 --2- 172.21.15.28:0/147002675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd54004d40 0x7efd54005150 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7efd60009f90 tx=0x7efd54005830 comp rx=0 tx=0).stop 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd6cfd1640 1 -- 172.21.15.28:0/147002675 reap_dead start 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd6ea34640 1 -- 172.21.15.28:0/147002675 shutdown_connections 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd6ea34640 1 -- 172.21.15.28:0/147002675 >> 172.21.15.28:0/147002675 conn(0x7efd680d2a50 msgr2=0x7efd680d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:13.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd6ea34640 1 -- 172.21.15.28:0/147002675 shutdown_connections 2024-04-25T19:12:13.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.996+0000 7efd6ea34640 1 -- 172.21.15.28:0/147002675 wait complete. 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.998+0000 7efd6ea34640 1 Processor -- start 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:12.998+0000 7efd6ea34640 1 -- start start 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6ea34640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd6814a650 0x7efd68159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6ea34640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68159ca0 0x7efd68146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6ea34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd681467a0 0x7efd680cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6ea34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd680d0bf0 con 0x7efd681467a0 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6ea34640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd680b9400 con 0x7efd6814a650 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6ea34640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efd680cc950 con 0x7efd68159ca0 2024-04-25T19:12:13.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6cfd1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd681467a0 0x7efd680cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6cfd1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd681467a0 0x7efd680cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56130/0 (socket says 172.21.15.28:56130) 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6cfd1640 1 -- 172.21.15.28:0/3831191505 learned_addr learned my addr 172.21.15.28:0/3831191505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd67fff640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd6814a650 0x7efd68159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd677fe640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68159ca0 0x7efd68146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6cfd1640 1 -- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68159ca0 msgr2=0x7efd68146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6cfd1640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68159ca0 0x7efd68146260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6cfd1640 1 -- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd6814a650 msgr2=0x7efd68159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6cfd1640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd6814a650 0x7efd68159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd6cfd1640 1 -- 172.21.15.28:0/3831191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd681451c0 con 0x7efd681467a0 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.000+0000 7efd67fff640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd6814a650 0x7efd68159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.001+0000 7efd6cfd1640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd681467a0 0x7efd680cdc10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd60009f90 tx=0x7efd60009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.001+0000 7efd64ff9640 1 -- 172.21.15.28:0/3831191505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60009c10 con 0x7efd681467a0 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.001+0000 7efd64ff9640 1 -- 172.21.15.28:0/3831191505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60031d70 con 0x7efd681467a0 2024-04-25T19:12:13.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.001+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd681454a0 con 0x7efd681467a0 2024-04-25T19:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.001+0000 7efd64ff9640 1 -- 172.21.15.28:0/3831191505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60003bd0 con 0x7efd681467a0 2024-04-25T19:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.001+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd68145a60 con 0x7efd681467a0 2024-04-25T19:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.002+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd68050740 con 0x7efd681467a0 2024-04-25T19:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.002+0000 7efd64ff9640 1 -- 172.21.15.28:0/3831191505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efd60005d10 con 0x7efd681467a0 2024-04-25T19:12:13.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.003+0000 7efd64ff9640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd480744e0 0x7efd480769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.003+0000 7efd67fff640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd480744e0 0x7efd480769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.003+0000 7efd64ff9640 1 -- 172.21.15.28:0/3831191505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7efd600b6930 con 0x7efd681467a0 2024-04-25T19:12:13.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.004+0000 7efd67fff640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd480744e0 0x7efd480769f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efd680d0960 tx=0x7efd50008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.008+0000 7efd64ff9640 1 -- 172.21.15.28:0/3831191505 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd6004a060 con 0x7efd681467a0 2024-04-25T19:12:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.192+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efd680cff90 con 0x7efd681467a0 2024-04-25T19:12:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.193+0000 7efd64ff9640 1 -- 172.21.15.28:0/3831191505 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7efd60080cb0 con 0x7efd681467a0 2024-04-25T19:12:13.195 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd480744e0 msgr2=0x7efd480769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd480744e0 0x7efd480769f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efd680d0960 tx=0x7efd50008040 comp rx=0 tx=0).stop 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd681467a0 msgr2=0x7efd680cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd681467a0 0x7efd680cdc10 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd60009f90 tx=0x7efd60009750 comp rx=0 tx=0).stop 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 shutdown_connections 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd480744e0 0x7efd480769f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd681467a0 0x7efd680cdc10 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd68159ca0 0x7efd68146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 --2- 172.21.15.28:0/3831191505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd6814a650 0x7efd68159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 >> 172.21.15.28:0/3831191505 conn(0x7efd680d2a50 msgr2=0x7efd680b9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:13.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 shutdown_connections 2024-04-25T19:12:13.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.195+0000 7efd6ea34640 1 -- 172.21.15.28:0/3831191505 wait complete. 2024-04-25T19:12:13.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:13.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.319+0000 7f75fcf0d640 1 Processor -- start 2024-04-25T19:12:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.319+0000 7f75fcf0d640 1 -- start start 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75fcf0d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f806e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75fcf0d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f75f80b39e0 0x7f75f80d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75fcf0d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f809d1a0 con 0x7f75f806e840 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75fcf0d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f75f806df50 con 0x7f75f80b39e0 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75fcf0d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f75f80d4870 con 0x7f75f809cd90 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75f659c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f806e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75f6d9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f75f80b39e0 0x7f75f80d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75f5d9b640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f75f806e840 0x7f75f80b32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:58822/0 (socket says 172.21.15.28:58822) 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.320+0000 7f75f5d9b640 1 -- 172.21.15.28:0/3365528193 learned_addr learned my addr 172.21.15.28:0/3365528193 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.321+0000 7f75f559a640 1 -- 172.21.15.28:0/3365528193 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 534524059 0 0) 0x7f75f809d1a0 con 0x7f75f806e840 2024-04-25T19:12:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.321+0000 7f75f559a640 1 -- 172.21.15.28:0/3365528193 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75e4003520 con 0x7f75f806e840 2024-04-25T19:12:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.321+0000 7f75f659c640 1 -- 172.21.15.28:0/3365528193 >> v2:172.21.15.53:3300/0 conn(0x7f75f80b39e0 msgr2=0x7f75f80d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.321+0000 7f75f659c640 1 --2- 172.21.15.28:0/3365528193 >> v2:172.21.15.53:3300/0 conn(0x7f75f80b39e0 0x7f75f80d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.321+0000 7f75f659c640 1 -- 172.21.15.28:0/3365528193 >> v1:172.21.15.53:6789/0 conn(0x7f75f806e840 legacy=0x7f75f80b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.321+0000 7f75f659c640 1 -- 172.21.15.28:0/3365528193 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f806d960 con 0x7f75f809cd90 2024-04-25T19:12:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.321+0000 7f75f6d9d640 1 --2- 172.21.15.28:0/3365528193 >> v2:172.21.15.53:3300/0 conn(0x7f75f80b39e0 0x7f75f80d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.322+0000 7f75f659c640 1 --2- 172.21.15.28:0/3365528193 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f806e300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75f00097a0 tx=0x7f75f002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d48e31fb34b18fb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.322+0000 7f75f559a640 1 -- 172.21.15.28:0/3365528193 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0037070 con 0x7f75f809cd90 2024-04-25T19:12:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.322+0000 7f75f559a640 1 -- 172.21.15.28:0/3365528193 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75f002ae50 con 0x7f75f809cd90 2024-04-25T19:12:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.322+0000 7f75f559a640 1 -- 172.21.15.28:0/3365528193 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75f0032640 con 0x7f75f809cd90 2024-04-25T19:12:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.323+0000 7f75fcf0d640 1 -- 172.21.15.28:0/3365528193 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 msgr2=0x7f75f806e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.323+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/3365528193 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f806e300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75f00097a0 tx=0x7f75f002a100 comp rx=0 tx=0).stop 2024-04-25T19:12:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.323+0000 7f75fcf0d640 1 -- 172.21.15.28:0/3365528193 shutdown_connections 2024-04-25T19:12:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.323+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/3365528193 >> v2:172.21.15.53:3300/0 conn(0x7f75f80b39e0 0x7f75f80d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.323+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/3365528193 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f806e300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.323+0000 7f75fcf0d640 1 -- 172.21.15.28:0/3365528193 >> 172.21.15.28:0/3365528193 conn(0x7f75f80d03f0 msgr2=0x7f75f80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.323+0000 7f75fcf0d640 1 -- 172.21.15.28:0/3365528193 shutdown_connections 2024-04-25T19:12:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.323+0000 7f75fcf0d640 1 -- 172.21.15.28:0/3365528193 wait complete. 2024-04-25T19:12:13.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.325+0000 7f75fcf0d640 1 Processor -- start 2024-04-25T19:12:13.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.325+0000 7f75fcf0d640 1 -- start start 2024-04-25T19:12:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75fcf0d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75f806e840 0x7f75f81e3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75fcf0d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f81df590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75fcf0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80b39e0 0x7f75f80b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75fcf0d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75f80b9180 con 0x7f75f80b39e0 2024-04-25T19:12:13.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75fcf0d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f75f806dc70 con 0x7f75f806e840 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75fcf0d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f75f80b9460 con 0x7f75f809cd90 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80b39e0 0x7f75f80b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75f6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80b39e0 0x7f75f80b1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56136/0 (socket says 172.21.15.28:56136) 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75f6d9d640 1 -- 172.21.15.28:0/2398787189 learned_addr learned my addr 172.21.15.28:0/2398787189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75f5d9b640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f81df590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.326+0000 7f75f659c640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75f806e840 0x7f75f81e3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75f6d9d640 1 -- 172.21.15.28:0/2398787189 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 msgr2=0x7f75f81df590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75f6d9d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f81df590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75f6d9d640 1 -- 172.21.15.28:0/2398787189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75f806e840 msgr2=0x7f75f81e3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75f6d9d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75f806e840 0x7f75f81e3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75f6d9d640 1 -- 172.21.15.28:0/2398787189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f80cc8f0 con 0x7f75f80b39e0 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75f6d9d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80b39e0 0x7f75f80b1d60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f75e800efc0 tx=0x7f75e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75eeffd640 1 -- 172.21.15.28:0/2398787189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8018030 con 0x7f75f80b39e0 2024-04-25T19:12:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75eeffd640 1 -- 172.21.15.28:0/2398787189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e800cdf0 con 0x7f75f80b39e0 2024-04-25T19:12:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f80b0cc0 con 0x7f75f80b39e0 2024-04-25T19:12:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75eeffd640 1 -- 172.21.15.28:0/2398787189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8003bb0 con 0x7f75f80b39e0 2024-04-25T19:12:13.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.327+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f80b1280 con 0x7f75f80b39e0 2024-04-25T19:12:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.328+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f80b0a30 con 0x7f75f80b39e0 2024-04-25T19:12:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.329+0000 7f75eeffd640 1 -- 172.21.15.28:0/2398787189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f75e8005d10 con 0x7f75f80b39e0 2024-04-25T19:12:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.329+0000 7f75eeffd640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f75d80744b0 0x7f75d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.329+0000 7f75eeffd640 1 -- 172.21.15.28:0/2398787189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f75e8095e20 con 0x7f75f80b39e0 2024-04-25T19:12:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.329+0000 7f75f659c640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f75d80744b0 0x7f75d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.330+0000 7f75f659c640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f75d80744b0 0x7f75d80769c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f75f0034000 tx=0x7f75f002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.334+0000 7f75eeffd640 1 -- 172.21.15.28:0/2398787189 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75e8022070 con 0x7f75f80b39e0 2024-04-25T19:12:13.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.523+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75f8050740 con 0x7f75f80b39e0 2024-04-25T19:12:13.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.524+0000 7f75eeffd640 1 -- 172.21.15.28:0/2398787189 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f75e8010040 con 0x7f75f80b39e0 2024-04-25T19:12:13.526 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:13.526 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:13.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f75d80744b0 msgr2=0x7f75d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f75d80744b0 0x7f75d80769c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f75f0034000 tx=0x7f75f002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80b39e0 msgr2=0x7f75f80b1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80b39e0 0x7f75f80b1d60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f75e800efc0 tx=0x7f75e800c490 comp rx=0 tx=0).stop 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 shutdown_connections 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f75d80744b0 0x7f75d80769c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75f80b39e0 0x7f75f80b1d60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f75f809cd90 0x7f75f81df590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 --2- 172.21.15.28:0/2398787189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75f806e840 0x7f75f81e3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.526+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 >> 172.21.15.28:0/2398787189 conn(0x7f75f80d03f0 msgr2=0x7f75f8072600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.527+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 shutdown_connections 2024-04-25T19:12:13.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.527+0000 7f75fcf0d640 1 -- 172.21.15.28:0/2398787189 wait complete. 2024-04-25T19:12:13.540 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:13.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:13.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.653+0000 7eff6db2d640 1 Processor -- start 2024-04-25T19:12:13.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.653+0000 7eff6db2d640 1 -- start start 2024-04-25T19:12:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff6db2d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b79f0 0x7eff680b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff6db2d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff68160720 con 0x7eff68160310 2024-04-25T19:12:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff6db2d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff680b8340 con 0x7eff680b9920 2024-04-25T19:12:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff6db2d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7eff680d0f40 con 0x7eff680b79f0 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff677fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7eff68160310 0x7eff680d5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50540/0 (socket says 172.21.15.28:50540) 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff677fe640 1 -- 172.21.15.28:0/3560749888 learned_addr learned my addr 172.21.15.28:0/3560749888 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff66ffd640 1 --2- 172.21.15.28:0/3560749888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b79f0 0x7eff680b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff667fc640 1 -- 172.21.15.28:0/3560749888 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1847967206 0 0) 0x7eff68160720 con 0x7eff68160310 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff667fc640 1 -- 172.21.15.28:0/3560749888 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff50003520 con 0x7eff68160310 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff667fc640 1 -- 172.21.15.28:0/3560749888 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1741343820 0 0) 0x7eff60001490 con 0x7eff680b9920 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff667fc640 1 -- 172.21.15.28:0/3560749888 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff68160720 con 0x7eff680b9920 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff66ffd640 1 -- 172.21.15.28:0/3560749888 >> v1:172.21.15.53:6789/0 conn(0x7eff680b9920 legacy=0x7eff680b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff667fc640 1 -- 172.21.15.28:0/3560749888 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2961459275 0 0) 0x7eff50003520 con 0x7eff68160310 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff66ffd640 1 -- 172.21.15.28:0/3560749888 >> v1:172.21.15.28:6789/0 conn(0x7eff68160310 legacy=0x7eff680d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.655+0000 7eff66ffd640 1 -- 172.21.15.28:0/3560749888 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff680d1220 con 0x7eff680b79f0 2024-04-25T19:12:13.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.656+0000 7eff66ffd640 1 --2- 172.21.15.28:0/3560749888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b79f0 0x7eff680b7e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0097d0 tx=0x7eff5c02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac4f47f360bf24d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.656+0000 7eff667fc640 1 -- 172.21.15.28:0/3560749888 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff5c037070 con 0x7eff680b79f0 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.656+0000 7eff667fc640 1 -- 172.21.15.28:0/3560749888 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff5c02ac30 con 0x7eff680b79f0 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.656+0000 7eff667fc640 1 -- 172.21.15.28:0/3560749888 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff58001670 con 0x7eff680b79f0 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.657+0000 7eff6db2d640 1 -- 172.21.15.28:0/3560749888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b79f0 msgr2=0x7eff680b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.657+0000 7eff6db2d640 1 --2- 172.21.15.28:0/3560749888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b79f0 0x7eff680b7e00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7eff5c0097d0 tx=0x7eff5c02a1a0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.657+0000 7eff6db2d640 1 -- 172.21.15.28:0/3560749888 shutdown_connections 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.657+0000 7eff6db2d640 1 --2- 172.21.15.28:0/3560749888 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b79f0 0x7eff680b7e00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.657+0000 7eff6db2d640 1 -- 172.21.15.28:0/3560749888 >> 172.21.15.28:0/3560749888 conn(0x7eff680d2a50 msgr2=0x7eff680d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.657+0000 7eff6db2d640 1 -- 172.21.15.28:0/3560749888 shutdown_connections 2024-04-25T19:12:13.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.657+0000 7eff6db2d640 1 -- 172.21.15.28:0/3560749888 wait complete. 2024-04-25T19:12:13.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.659+0000 7eff6db2d640 1 Processor -- start 2024-04-25T19:12:13.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.659+0000 7eff6db2d640 1 -- start start 2024-04-25T19:12:13.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff6db2d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff680b79f0 0x7eff6814e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff6db2d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b9920 0x7eff681f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff6db2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68160310 0x7eff680b6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff6db2d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff680d0bf0 con 0x7eff68160310 2024-04-25T19:12:13.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff6db2d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7eff680b9400 con 0x7eff680b79f0 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff6db2d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7eff680cc950 con 0x7eff680b9920 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff67fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68160310 0x7eff680b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff67fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68160310 0x7eff680b6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56146/0 (socket says 172.21.15.28:56146) 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.660+0000 7eff67fff640 1 -- 172.21.15.28:0/3051950434 learned_addr learned my addr 172.21.15.28:0/3051950434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff66ffd640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b9920 0x7eff681f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff677fe640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff680b79f0 0x7eff6814e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff67fff640 1 -- 172.21.15.28:0/3051950434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b9920 msgr2=0x7eff681f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff67fff640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b9920 0x7eff681f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff67fff640 1 -- 172.21.15.28:0/3051950434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff680b79f0 msgr2=0x7eff6814e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff67fff640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff680b79f0 0x7eff6814e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff67fff640 1 -- 172.21.15.28:0/3051950434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff681583f0 con 0x7eff68160310 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff67fff640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68160310 0x7eff680b6630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7eff6000c770 tx=0x7eff6000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff4ffff640 1 -- 172.21.15.28:0/3051950434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff60003b70 con 0x7eff68160310 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff4ffff640 1 -- 172.21.15.28:0/3051950434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff60003d10 con 0x7eff68160310 2024-04-25T19:12:13.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff4ffff640 1 -- 172.21.15.28:0/3051950434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6000a9f0 con 0x7eff68160310 2024-04-25T19:12:13.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff6815a5c0 con 0x7eff68160310 2024-04-25T19:12:13.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.661+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff68157b40 con 0x7eff68160310 2024-04-25T19:12:13.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.662+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff68158160 con 0x7eff68160310 2024-04-25T19:12:13.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.663+0000 7eff4ffff640 1 -- 172.21.15.28:0/3051950434 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7eff6000ab90 con 0x7eff68160310 2024-04-25T19:12:13.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.663+0000 7eff4ffff640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7eff540744e0 0x7eff540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.664+0000 7eff4ffff640 1 -- 172.21.15.28:0/3051950434 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7eff600953a0 con 0x7eff68160310 2024-04-25T19:12:13.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.664+0000 7eff677fe640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7eff540744e0 0x7eff540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.664+0000 7eff677fe640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7eff540744e0 0x7eff540769f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7eff580017e0 tx=0x7eff58001610 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.668+0000 7eff4ffff640 1 -- 172.21.15.28:0/3051950434 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7eff680b8340 con 0x7eff68160310 2024-04-25T19:12:13.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.851+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff68050740 con 0x7eff68160310 2024-04-25T19:12:13.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.852+0000 7eff4ffff640 1 -- 172.21.15.28:0/3051950434 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7eff6005f0d0 con 0x7eff68160310 2024-04-25T19:12:13.854 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.853+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7eff540744e0 msgr2=0x7eff540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.853+0000 7eff6db2d640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7eff540744e0 0x7eff540769f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7eff580017e0 tx=0x7eff58001610 comp rx=0 tx=0).stop 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.853+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68160310 msgr2=0x7eff680b6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.853+0000 7eff6db2d640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68160310 0x7eff680b6630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7eff6000c770 tx=0x7eff6000cc30 comp rx=0 tx=0).stop 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.854+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 shutdown_connections 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.854+0000 7eff6db2d640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7eff540744e0 0x7eff540769f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.854+0000 7eff6db2d640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff68160310 0x7eff680b6630 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.854+0000 7eff6db2d640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7eff680b9920 0x7eff681f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.854+0000 7eff6db2d640 1 --2- 172.21.15.28:0/3051950434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff680b79f0 0x7eff6814e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.854+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 >> 172.21.15.28:0/3051950434 conn(0x7eff680d2a50 msgr2=0x7eff680bac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:13.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.854+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 shutdown_connections 2024-04-25T19:12:13.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.854+0000 7eff6db2d640 1 -- 172.21.15.28:0/3051950434 wait complete. 2024-04-25T19:12:13.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:13.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.981+0000 7fa1436b1640 1 Processor -- start 2024-04-25T19:12:13.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.981+0000 7fa1436b1640 1 -- start start 2024-04-25T19:12:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.982+0000 7fa1436b1640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa13c164990 0x7fa13c15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.982+0000 7fa1436b1640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa13c161210 con 0x7fa13c0b7660 2024-04-25T19:12:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.982+0000 7fa1436b1640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa13c1614a0 con 0x7fa13c1606f0 2024-04-25T19:12:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.982+0000 7fa1436b1640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fa13c163eb0 con 0x7fa13c164990 2024-04-25T19:12:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.982+0000 7fa14144d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa13c164990 0x7fa13c15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.982+0000 7fa140c4c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa13c0b7660 0x7fa13c0b7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50542/0 (socket says 172.21.15.28:50542) 2024-04-25T19:12:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.983+0000 7fa140c4c640 1 -- 172.21.15.28:0/1852947936 learned_addr learned my addr 172.21.15.28:0/1852947936 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:13.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.983+0000 7fa14144d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa13c164990 0x7fa13c15dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56156/0 (socket says 172.21.15.28:56156) 2024-04-25T19:12:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.983+0000 7fa14144d640 1 -- 172.21.15.28:0/1852947936 >> v1:172.21.15.53:6789/0 conn(0x7fa13c1606f0 legacy=0x7fa13c160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.983+0000 7fa14144d640 1 -- 172.21.15.28:0/1852947936 >> v1:172.21.15.28:6789/0 conn(0x7fa13c0b7660 legacy=0x7fa13c0b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.983+0000 7fa133fff640 1 -- 172.21.15.28:0/1852947936 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 474435558 0 0) 0x7fa13c161210 con 0x7fa13c0b7660 2024-04-25T19:12:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.983+0000 7fa14144d640 1 -- 172.21.15.28:0/1852947936 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13c164190 con 0x7fa13c164990 2024-04-25T19:12:13.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa14144d640 1 --2- 172.21.15.28:0/1852947936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c164990 0x7fa13c15dcb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa1380097a0 tx=0x7fa13802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891ca1bcc90c8e2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa133fff640 1 -- 172.21.15.28:0/1852947936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138037070 con 0x7fa13c164990 2024-04-25T19:12:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa133fff640 1 -- 172.21.15.28:0/1852947936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa138032440 con 0x7fa13c164990 2024-04-25T19:12:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa133fff640 1 -- 172.21.15.28:0/1852947936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138032810 con 0x7fa13c164990 2024-04-25T19:12:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa1436b1640 1 -- 172.21.15.28:0/1852947936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c164990 msgr2=0x7fa13c15dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa1436b1640 1 --2- 172.21.15.28:0/1852947936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c164990 0x7fa13c15dcb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa1380097a0 tx=0x7fa13802a230 comp rx=0 tx=0).stop 2024-04-25T19:12:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa1436b1640 1 -- 172.21.15.28:0/1852947936 shutdown_connections 2024-04-25T19:12:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa1436b1640 1 --2- 172.21.15.28:0/1852947936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c164990 0x7fa13c15dcb0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.984+0000 7fa1436b1640 1 -- 172.21.15.28:0/1852947936 >> 172.21.15.28:0/1852947936 conn(0x7fa13c0d2840 msgr2=0x7fa13c0d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.985+0000 7fa1436b1640 1 -- 172.21.15.28:0/1852947936 shutdown_connections 2024-04-25T19:12:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.985+0000 7fa1436b1640 1 -- 172.21.15.28:0/1852947936 wait complete. 2024-04-25T19:12:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.986+0000 7fa1436b1640 1 Processor -- start 2024-04-25T19:12:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.986+0000 7fa1436b1640 1 -- start start 2024-04-25T19:12:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa1436b1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa13c0b7660 0x7fa13c156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa1436b1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c1606f0 0x7fa13c1f9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa1436b1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa13c164990 0x7fa13c1f9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa1436b1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c0b4750 con 0x7fa13c1606f0 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa1436b1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa13c0b45d0 con 0x7fa13c164990 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa1436b1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa13c0b48d0 con 0x7fa13c0b7660 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c1606f0 0x7fa13c1f9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c1606f0 0x7fa13c1f9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56162/0 (socket says 172.21.15.28:56162) 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 -- 172.21.15.28:0/2946549528 learned_addr learned my addr 172.21.15.28:0/2946549528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa14144d640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa13c0b7660 0x7fa13c156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa141c4e640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa13c164990 0x7fa13c1f9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 -- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa13c0b7660 msgr2=0x7fa13c156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa13c0b7660 0x7fa13c156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 -- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa13c164990 msgr2=0x7fa13c1f9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa13c164990 0x7fa13c1f9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 -- 172.21.15.28:0/2946549528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13c1eb860 con 0x7fa13c1606f0 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa14144d640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa13c0b7660 0x7fa13c156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa140c4c640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c1606f0 0x7fa13c1f9930 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa12c00c3f0 tx=0x7fa12c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa131ffb640 1 -- 172.21.15.28:0/2946549528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c0074e0 con 0x7fa13c1606f0 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa131ffb640 1 -- 172.21.15.28:0/2946549528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12c0047e0 con 0x7fa13c1606f0 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa131ffb640 1 -- 172.21.15.28:0/2946549528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c004bd0 con 0x7fa13c1606f0 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.988+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13c1ebb40 con 0x7fa13c1606f0 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.989+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa13c15e1e0 con 0x7fa13c1606f0 2024-04-25T19:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.990+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa13c050740 con 0x7fa13c1606f0 2024-04-25T19:12:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.990+0000 7fa131ffb640 1 -- 172.21.15.28:0/2946549528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa12c011070 con 0x7fa13c1606f0 2024-04-25T19:12:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.991+0000 7fa131ffb640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.991+0000 7fa131ffb640 1 -- 172.21.15.28:0/2946549528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa12c095190 con 0x7fa13c1606f0 2024-04-25T19:12:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.991+0000 7fa14144d640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.991+0000 7fa14144d640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa138034000 tx=0x7fa13802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:13.995+0000 7fa131ffb640 1 -- 172.21.15.28:0/2946549528 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa12c05f5c0 con 0x7fa13c1606f0 2024-04-25T19:12:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.182+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa13c0d4000 con 0x7fa13c1606f0 2024-04-25T19:12:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.182+0000 7fa131ffb640 1 -- 172.21.15.28:0/2946549528 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fa12c05eec0 con 0x7fa13c1606f0 2024-04-25T19:12:14.184 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 msgr2=0x7fa1200769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa138034000 tx=0x7fa13802e000 comp rx=0 tx=0).stop 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c1606f0 msgr2=0x7fa13c1f9930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c1606f0 0x7fa13c1f9930 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa12c00c3f0 tx=0x7fa12c00c8b0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 shutdown_connections 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa1200744e0 0x7fa1200769f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa13c164990 0x7fa13c1f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa13c1606f0 0x7fa13c1f9930 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 --2- 172.21.15.28:0/2946549528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa13c0b7660 0x7fa13c156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 >> 172.21.15.28:0/2946549528 conn(0x7fa13c0d2840 msgr2=0x7fa13c0b7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 shutdown_connections 2024-04-25T19:12:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.184+0000 7fa1436b1640 1 -- 172.21.15.28:0/2946549528 wait complete. 2024-04-25T19:12:14.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.310+0000 7f4d1f91c640 1 Processor -- start 2024-04-25T19:12:14.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.311+0000 7f4d1f91c640 1 -- start start 2024-04-25T19:12:14.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d1f91c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d180ba930 0x7f4d1816ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d1f91c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d18160720 con 0x7f4d18160310 2024-04-25T19:12:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d1f91c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d180d0f40 con 0x7f4d180b79f0 2024-04-25T19:12:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d1f91c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4d180d1220 con 0x7f4d180ba930 2024-04-25T19:12:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d1d6b8640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4d18160310 0x7f4d180d5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50558/0 (socket says 172.21.15.28:50558) 2024-04-25T19:12:14.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d1d6b8640 1 -- 172.21.15.28:0/1273674253 learned_addr learned my addr 172.21.15.28:0/1273674253 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194981203 0 0) 0x7f4d18160720 con 0x7f4d18160310 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d00003520 con 0x7f4d18160310 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.312+0000 7f4d1deb9640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d180ba930 0x7f4d1816ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3123925148 0 0) 0x7f4d00003520 con 0x7f4d18160310 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d180ba930 msgr2=0x7f4d1816ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d180ba930 0x7f4d1816ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 >> v1:172.21.15.53:6789/0 conn(0x7f4d180b79f0 legacy=0x7f4d180b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d180d1500 con 0x7f4d18160310 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f4d10001670 con 0x7f4d18160310 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 >> v1:172.21.15.28:6789/0 conn(0x7f4d18160310 legacy=0x7f4d180d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d000039e0 0x7f4d00003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d180ba930 0x7f4d00004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d00004d40 0x7f4d00005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d180d1220 con 0x7f4d00004d40 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4d0000baf0 con 0x7f4d000039e0 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4d0000bcf0 con 0x7f4d180ba930 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d1deb9640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d00004d40 0x7f4d00005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d1d6b8640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d000039e0 0x7f4d00003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d1ceb7640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d180ba930 0x7f4d00004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d1deb9640 1 -- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d180ba930 msgr2=0x7f4d00004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d1deb9640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d180ba930 0x7f4d00004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d1deb9640 1 -- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d000039e0 msgr2=0x7f4d00003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d1deb9640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d000039e0 0x7f4d00003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.313+0000 7f4d1deb9640 1 -- 172.21.15.28:0/1273674253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d00003520 con 0x7f4d00004d40 2024-04-25T19:12:14.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.314+0000 7f4d1deb9640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d00004d40 0x7f4d00005150 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009fc0 tx=0x7f4d00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fba2a696f86ae61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.314+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08034020 con 0x7f4d00004d40 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.314+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d080305f0 con 0x7f4d00004d40 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.314+0000 7f4d17fff640 1 -- 172.21.15.28:0/1273674253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d08009c10 con 0x7f4d00004d40 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.315+0000 7f4d1f91c640 1 -- 172.21.15.28:0/1273674253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d00004d40 msgr2=0x7f4d00005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.315+0000 7f4d1f91c640 1 --2- 172.21.15.28:0/1273674253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d00004d40 0x7f4d00005150 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009fc0 tx=0x7f4d00005830 comp rx=0 tx=0).stop 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.315+0000 7f4d1deb9640 1 -- 172.21.15.28:0/1273674253 reap_dead start 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.315+0000 7f4d1f91c640 1 -- 172.21.15.28:0/1273674253 shutdown_connections 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.315+0000 7f4d1f91c640 1 -- 172.21.15.28:0/1273674253 >> 172.21.15.28:0/1273674253 conn(0x7f4d180d2a50 msgr2=0x7f4d180d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.315+0000 7f4d1f91c640 1 -- 172.21.15.28:0/1273674253 shutdown_connections 2024-04-25T19:12:14.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.315+0000 7f4d1f91c640 1 -- 172.21.15.28:0/1273674253 wait complete. 2024-04-25T19:12:14.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.317+0000 7f4d1f91c640 1 Processor -- start 2024-04-25T19:12:14.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.317+0000 7f4d1f91c640 1 -- start start 2024-04-25T19:12:14.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1f91c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d00004d40 0x7f4d180cf380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1f91c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d181f5bd0 0x7f4d1814b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1f91c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d1814bea0 0x7f4d18158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1f91c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d180cc950 con 0x7f4d181f5bd0 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1f91c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4d180d0bf0 con 0x7f4d00004d40 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1f91c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4d180d17e0 con 0x7f4d1814bea0 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1ceb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d181f5bd0 0x7f4d1814b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1ceb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d181f5bd0 0x7f4d1814b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56178/0 (socket says 172.21.15.28:56178) 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1ceb7640 1 -- 172.21.15.28:0/4242361325 learned_addr learned my addr 172.21.15.28:0/4242361325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1d6b8640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d00004d40 0x7f4d180cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1deb9640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d1814bea0 0x7f4d18158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1ceb7640 1 -- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d1814bea0 msgr2=0x7f4d18158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1ceb7640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d1814bea0 0x7f4d18158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1ceb7640 1 -- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d00004d40 msgr2=0x7f4d180cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1ceb7640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d00004d40 0x7f4d180cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1ceb7640 1 -- 172.21.15.28:0/4242361325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d1814c2b0 con 0x7f4d181f5bd0 2024-04-25T19:12:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.318+0000 7f4d1d6b8640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d00004d40 0x7f4d180cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.319+0000 7f4d1ceb7640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d181f5bd0 0x7f4d1814b960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00f640 tx=0x7f4d0c00fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.319+0000 7f4d15ffb640 1 -- 172.21.15.28:0/4242361325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c0052e0 con 0x7f4d181f5bd0 2024-04-25T19:12:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.319+0000 7f4d15ffb640 1 -- 172.21.15.28:0/4242361325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0c018070 con 0x7f4d181f5bd0 2024-04-25T19:12:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.319+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d1814cdc0 con 0x7f4d181f5bd0 2024-04-25T19:12:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.319+0000 7f4d15ffb640 1 -- 172.21.15.28:0/4242361325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c003830 con 0x7f4d181f5bd0 2024-04-25T19:12:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.319+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d180ce250 con 0x7f4d181f5bd0 2024-04-25T19:12:14.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.320+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d18050740 con 0x7f4d181f5bd0 2024-04-25T19:12:14.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.320+0000 7f4d15ffb640 1 -- 172.21.15.28:0/4242361325 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4d0c0039d0 con 0x7f4d181f5bd0 2024-04-25T19:12:14.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.321+0000 7f4d15ffb640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4d040744e0 0x7f4d040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.321+0000 7f4d15ffb640 1 -- 172.21.15.28:0/4242361325 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4d0c095360 con 0x7f4d181f5bd0 2024-04-25T19:12:14.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.321+0000 7f4d1d6b8640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4d040744e0 0x7f4d040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.321+0000 7f4d1d6b8640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4d040744e0 0x7f4d040769f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4d10007d70 tx=0x7f4d10009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.326+0000 7f4d15ffb640 1 -- 172.21.15.28:0/4242361325 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d1814c2b0 con 0x7f4d181f5bd0 2024-04-25T19:12:14.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.511+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4d180ce420 con 0x7f4d181f5bd0 2024-04-25T19:12:14.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.512+0000 7f4d15ffb640 1 -- 172.21.15.28:0/4242361325 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f4d0c05f6f0 con 0x7f4d181f5bd0 2024-04-25T19:12:14.514 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:14.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4d040744e0 msgr2=0x7f4d040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4d040744e0 0x7f4d040769f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4d10007d70 tx=0x7f4d10009040 comp rx=0 tx=0).stop 2024-04-25T19:12:14.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d181f5bd0 msgr2=0x7f4d1814b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d181f5bd0 0x7f4d1814b960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00f640 tx=0x7f4d0c00fb00 comp rx=0 tx=0).stop 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 shutdown_connections 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4d040744e0 0x7f4d040769f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4d1814bea0 0x7f4d18158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d181f5bd0 0x7f4d1814b960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 --2- 172.21.15.28:0/4242361325 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4d00004d40 0x7f4d180cf380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 >> 172.21.15.28:0/4242361325 conn(0x7f4d180d2a50 msgr2=0x7f4d180d5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 shutdown_connections 2024-04-25T19:12:14.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.514+0000 7f4d1f91c640 1 -- 172.21.15.28:0/4242361325 wait complete. 2024-04-25T19:12:14.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:14.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.640+0000 7f4dc87ab640 1 Processor -- start 2024-04-25T19:12:14.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.640+0000 7f4dc87ab640 1 -- start start 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc87ab640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4dc015aaa0 0x7f4dc00affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc87ab640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc006f5b0 con 0x7f4dc00d0ee0 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc87ab640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc00d14a0 con 0x7f4dc00b5020 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc87ab640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc00af290 con 0x7f4dc015aaa0 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc6547640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4dc015aaa0 0x7f4dc00affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc5d46640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4dc00d0ee0 0x7f4dc00b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50562/0 (socket says 172.21.15.28:50562) 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc6547640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4dc015aaa0 0x7f4dc00affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56186/0 (socket says 172.21.15.28:56186) 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc5d46640 1 -- 172.21.15.28:0/771360254 learned_addr learned my addr 172.21.15.28:0/771360254 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc6547640 1 -- 172.21.15.28:0/771360254 >> v1:172.21.15.53:6789/0 conn(0x7f4dc00b5020 legacy=0x7f4dc0160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:14.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc6547640 1 -- 172.21.15.28:0/771360254 >> v1:172.21.15.28:6789/0 conn(0x7f4dc00d0ee0 legacy=0x7f4dc00b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc6547640 1 -- 172.21.15.28:0/771360254 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc00af4e0 con 0x7f4dc015aaa0 2024-04-25T19:12:14.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.642+0000 7f4dc5545640 1 -- 172.21.15.28:0/771360254 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911856430 0 0) 0x7f4dc006f5b0 con 0x7f4dc00d0ee0 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.643+0000 7f4dc6547640 1 --2- 172.21.15.28:0/771360254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc015aaa0 0x7f4dc00affc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009d90 tx=0x7f4db002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a88d15aeed8e30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.643+0000 7f4dc5545640 1 -- 172.21.15.28:0/771360254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0007740 con 0x7f4dc015aaa0 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.643+0000 7f4dc5545640 1 -- 172.21.15.28:0/771360254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db00078e0 con 0x7f4dc015aaa0 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.643+0000 7f4dc5545640 1 -- 172.21.15.28:0/771360254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0007cb0 con 0x7f4dc015aaa0 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.644+0000 7f4dc87ab640 1 -- 172.21.15.28:0/771360254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc015aaa0 msgr2=0x7f4dc00affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.644+0000 7f4dc87ab640 1 --2- 172.21.15.28:0/771360254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc015aaa0 0x7f4dc00affc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009d90 tx=0x7f4db002a6a0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.644+0000 7f4dc87ab640 1 -- 172.21.15.28:0/771360254 shutdown_connections 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.644+0000 7f4dc87ab640 1 --2- 172.21.15.28:0/771360254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc015aaa0 0x7f4dc00affc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.644+0000 7f4dc87ab640 1 -- 172.21.15.28:0/771360254 >> 172.21.15.28:0/771360254 conn(0x7f4dc00742c0 msgr2=0x7f4dc006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.644+0000 7f4dc87ab640 1 -- 172.21.15.28:0/771360254 shutdown_connections 2024-04-25T19:12:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.644+0000 7f4dc87ab640 1 -- 172.21.15.28:0/771360254 wait complete. 2024-04-25T19:12:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.646+0000 7f4dc87ab640 1 Processor -- start 2024-04-25T19:12:14.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.646+0000 7f4dc87ab640 1 -- start start 2024-04-25T19:12:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc87ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc00b5020 0x7f4dc01591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc87ab640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4dc00d0ee0 0x7f4dc0159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc87ab640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc015aaa0 0x7f4dc0158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc87ab640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc00b5430 con 0x7f4dc00b5020 2024-04-25T19:12:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc87ab640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc00afaa0 con 0x7f4dc015aaa0 2024-04-25T19:12:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc87ab640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc00b55b0 con 0x7f4dc00d0ee0 2024-04-25T19:12:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6547640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc00b5020 0x7f4dc01591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6547640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc00b5020 0x7f4dc01591c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56200/0 (socket says 172.21.15.28:56200) 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6547640 1 -- 172.21.15.28:0/1389754924 learned_addr learned my addr 172.21.15.28:0/1389754924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc5d46640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4dc00d0ee0 0x7f4dc0159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6d48640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc015aaa0 0x7f4dc0158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6547640 1 -- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4dc00d0ee0 msgr2=0x7f4dc0159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6547640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4dc00d0ee0 0x7f4dc0159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6547640 1 -- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc015aaa0 msgr2=0x7f4dc0158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6547640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc015aaa0 0x7f4dc0158120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6547640 1 -- 172.21.15.28:0/1389754924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc009bba0 con 0x7f4dc00b5020 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc5d46640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4dc00d0ee0 0x7f4dc0159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.647+0000 7f4dc6d48640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc015aaa0 0x7f4dc0158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.648+0000 7f4dc6547640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc00b5020 0x7f4dc01591c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009eb0 tx=0x7f4db0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.648+0000 7f4dbeffd640 1 -- 172.21.15.28:0/1389754924 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0040070 con 0x7f4dc00b5020 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.648+0000 7f4dbeffd640 1 -- 172.21.15.28:0/1389754924 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db0031c40 con 0x7f4dc00b5020 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.648+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc009be80 con 0x7f4dc00b5020 2024-04-25T19:12:14.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.648+0000 7f4dbeffd640 1 -- 172.21.15.28:0/1389754924 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db003c420 con 0x7f4dc00b5020 2024-04-25T19:12:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.648+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc009c440 con 0x7f4dc00b5020 2024-04-25T19:12:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.649+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc0050740 con 0x7f4dc00b5020 2024-04-25T19:12:14.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.650+0000 7f4dbeffd640 1 -- 172.21.15.28:0/1389754924 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4db0039040 con 0x7f4dc00b5020 2024-04-25T19:12:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.650+0000 7f4dbeffd640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dac0744b0 0x7f4dac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.650+0000 7f4dbeffd640 1 -- 172.21.15.28:0/1389754924 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4db00b6cc0 con 0x7f4dc00b5020 2024-04-25T19:12:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.650+0000 7f4dc5d46640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dac0744b0 0x7f4dac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.651+0000 7f4dc5d46640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dac0744b0 0x7f4dac0769c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4dc015e250 tx=0x7f4db4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.655+0000 7f4dbeffd640 1 -- 172.21.15.28:0/1389754924 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4db0081050 con 0x7f4dc00b5020 2024-04-25T19:12:14.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.840+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4dc0070f20 con 0x7f4dc00b5020 2024-04-25T19:12:14.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.840+0000 7f4dbeffd640 1 -- 172.21.15.28:0/1389754924 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f4db0080950 con 0x7f4dc00b5020 2024-04-25T19:12:14.842 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dac0744b0 msgr2=0x7f4dac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dac0744b0 0x7f4dac0769c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4dc015e250 tx=0x7f4db4009040 comp rx=0 tx=0).stop 2024-04-25T19:12:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc00b5020 msgr2=0x7f4dc01591c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc00b5020 0x7f4dc01591c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4db0009eb0 tx=0x7f4db0004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 shutdown_connections 2024-04-25T19:12:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dac0744b0 0x7f4dac0769c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dc015aaa0 0x7f4dc0158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4dc00d0ee0 0x7f4dc0159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 --2- 172.21.15.28:0/1389754924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dc00b5020 0x7f4dc01591c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 >> 172.21.15.28:0/1389754924 conn(0x7f4dc00742c0 msgr2=0x7f4dc015d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 shutdown_connections 2024-04-25T19:12:14.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.842+0000 7f4dc87ab640 1 -- 172.21.15.28:0/1389754924 wait complete. 2024-04-25T19:12:14.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:14.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.966+0000 7f1781ae7640 1 Processor -- start 2024-04-25T19:12:14.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.966+0000 7f1781ae7640 1 -- start start 2024-04-25T19:12:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f1781ae7640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f177c0b5130 0x7f177c161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f1781ae7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f177c163e80 con 0x7f177c06ded0 2024-04-25T19:12:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f1781ae7640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f177c164160 con 0x7f177c15f7d0 2024-04-25T19:12:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f1781ae7640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f177c163c30 con 0x7f177c0b5130 2024-04-25T19:12:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f177affd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f177c06ded0 0x7f177c06e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50564/0 (socket says 172.21.15.28:50564) 2024-04-25T19:12:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f177affd640 1 -- 172.21.15.28:0/1579403758 learned_addr learned my addr 172.21.15.28:0/1579403758 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f177b7fe640 1 --2- 172.21.15.28:0/1579403758 >> v2:172.21.15.53:3300/0 conn(0x7f177c0b5130 0x7f177c161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f177a7fc640 1 -- 172.21.15.28:0/1579403758 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1366331457 0 0) 0x7f177c163e80 con 0x7f177c06ded0 2024-04-25T19:12:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.967+0000 7f177a7fc640 1 -- 172.21.15.28:0/1579403758 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1764003520 con 0x7f177c06ded0 2024-04-25T19:12:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.968+0000 7f177b7fe640 1 -- 172.21.15.28:0/1579403758 >> v1:172.21.15.53:6789/0 conn(0x7f177c15f7d0 legacy=0x7f177c15fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.968+0000 7f177b7fe640 1 -- 172.21.15.28:0/1579403758 >> v1:172.21.15.28:6789/0 conn(0x7f177c06ded0 legacy=0x7f177c06e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.968+0000 7f177b7fe640 1 -- 172.21.15.28:0/1579403758 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c164440 con 0x7f177c0b5130 2024-04-25T19:12:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.968+0000 7f177b7fe640 1 --2- 172.21.15.28:0/1579403758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 0x7f177c161130 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f176c0097a0 tx=0x7f176c02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d26330df12b1a5c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f177a7fc640 1 -- 172.21.15.28:0/1579403758 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c037070 con 0x7f177c0b5130 2024-04-25T19:12:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f177a7fc640 1 -- 172.21.15.28:0/1579403758 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176c032440 con 0x7f177c0b5130 2024-04-25T19:12:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f177a7fc640 1 -- 172.21.15.28:0/1579403758 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c0327f0 con 0x7f177c0b5130 2024-04-25T19:12:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f1781ae7640 1 -- 172.21.15.28:0/1579403758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 msgr2=0x7f177c161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f1781ae7640 1 --2- 172.21.15.28:0/1579403758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 0x7f177c161130 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f176c0097a0 tx=0x7f176c02a230 comp rx=0 tx=0).stop 2024-04-25T19:12:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f1781ae7640 1 -- 172.21.15.28:0/1579403758 shutdown_connections 2024-04-25T19:12:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f1781ae7640 1 --2- 172.21.15.28:0/1579403758 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 0x7f177c161130 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f1781ae7640 1 -- 172.21.15.28:0/1579403758 >> 172.21.15.28:0/1579403758 conn(0x7f177c0b7ad0 msgr2=0x7f177c0b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.969+0000 7f1781ae7640 1 -- 172.21.15.28:0/1579403758 shutdown_connections 2024-04-25T19:12:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.970+0000 7f1781ae7640 1 -- 172.21.15.28:0/1579403758 wait complete. 2024-04-25T19:12:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.971+0000 7f1781ae7640 1 Processor -- start 2024-04-25T19:12:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.971+0000 7f1781ae7640 1 -- start start 2024-04-25T19:12:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f1781ae7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f177c06ded0 0x7f177c1f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f1781ae7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 0x7f177c1f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f1781ae7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f177c15f7d0 0x7f177c15c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f1781ae7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f177c0d4910 con 0x7f177c06ded0 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f1781ae7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f177c164a90 con 0x7f177c0b5130 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f1781ae7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f177c0d4a90 con 0x7f177c15f7d0 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f177c06ded0 0x7f177c1f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f177c06ded0 0x7f177c1f7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56212/0 (socket says 172.21.15.28:56212) 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 -- 172.21.15.28:0/322979464 learned_addr learned my addr 172.21.15.28:0/322979464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177bfff640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f177c15f7d0 0x7f177c15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177affd640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 0x7f177c1f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 -- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f177c15f7d0 msgr2=0x7f177c15c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f177c15f7d0 0x7f177c15c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 -- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 msgr2=0x7f177c1f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 0x7f177c1f6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:14.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 -- 172.21.15.28:0/322979464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c1f7ed0 con 0x7f177c06ded0 2024-04-25T19:12:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177affd640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 0x7f177c1f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177bfff640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f177c15f7d0 0x7f177c15c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f177b7fe640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f177c06ded0 0x7f177c1f7740 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f176c0097a0 tx=0x7f176c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.973+0000 7f1763fff640 1 -- 172.21.15.28:0/322979464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c037050 con 0x7f177c06ded0 2024-04-25T19:12:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.974+0000 7f1763fff640 1 -- 172.21.15.28:0/322979464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176c038040 con 0x7f177c06ded0 2024-04-25T19:12:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.974+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177c1f7c80 con 0x7f177c06ded0 2024-04-25T19:12:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.974+0000 7f1763fff640 1 -- 172.21.15.28:0/322979464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c004470 con 0x7f177c06ded0 2024-04-25T19:12:14.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.974+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177c1f83d0 con 0x7f177c06ded0 2024-04-25T19:12:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.974+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f177c0d2a20 con 0x7f177c06ded0 2024-04-25T19:12:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.975+0000 7f1763fff640 1 -- 172.21.15.28:0/322979464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f176c03c6d0 con 0x7f177c06ded0 2024-04-25T19:12:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.976+0000 7f1763fff640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f17680744b0 0x7f17680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.976+0000 7f1763fff640 1 -- 172.21.15.28:0/322979464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f176c0b60c0 con 0x7f177c06ded0 2024-04-25T19:12:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.976+0000 7f177affd640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f17680744b0 0x7f17680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.976+0000 7f177affd640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f17680744b0 0x7f17680769c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1770007fd0 tx=0x7f1770009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:14.980+0000 7f1763fff640 1 -- 172.21.15.28:0/322979464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f176c080450 con 0x7f177c06ded0 2024-04-25T19:12:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.168+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f177c050740 con 0x7f177c06ded0 2024-04-25T19:12:15.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.169+0000 7f1763fff640 1 -- 172.21.15.28:0/322979464 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f176c07fd50 con 0x7f177c06ded0 2024-04-25T19:12:15.171 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:15.172 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f17680744b0 msgr2=0x7f17680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f17680744b0 0x7f17680769c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1770007fd0 tx=0x7f1770009040 comp rx=0 tx=0).stop 2024-04-25T19:12:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f177c06ded0 msgr2=0x7f177c1f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f177c06ded0 0x7f177c1f7740 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f176c0097a0 tx=0x7f176c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 shutdown_connections 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f17680744b0 0x7f17680769c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f177c15f7d0 0x7f177c15c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f177c0b5130 0x7f177c1f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 --2- 172.21.15.28:0/322979464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f177c06ded0 0x7f177c1f7740 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.171+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 >> 172.21.15.28:0/322979464 conn(0x7f177c0b7ad0 msgr2=0x7f177c0b5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.172+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 shutdown_connections 2024-04-25T19:12:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.172+0000 7f1781ae7640 1 -- 172.21.15.28:0/322979464 wait complete. 2024-04-25T19:12:15.185 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:15.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.298+0000 7f5d33630640 1 Processor -- start 2024-04-25T19:12:15.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.299+0000 7f5d33630640 1 -- start start 2024-04-25T19:12:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d33630640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5d2c164990 0x7f5d2c15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d33630640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d2c0b7660 0x7f5d2c0b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d33630640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d2c161210 con 0x7f5d2c1606f0 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d33630640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f5d2c0b7fb0 con 0x7f5d2c0b7660 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d33630640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5d2c1614a0 con 0x7f5d2c164990 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d30bcb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d2c0b7660 0x7f5d2c0b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d30bcb640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d2c0b7660 0x7f5d2c0b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56226/0 (socket says 172.21.15.28:56226) 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d30bcb640 1 -- 172.21.15.28:0/3502908978 learned_addr learned my addr 172.21.15.28:0/3502908978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.300+0000 7f5d313cc640 1 --2- 172.21.15.28:0/3502908978 >> v2:172.21.15.53:3300/0 conn(0x7f5d2c164990 0x7f5d2c15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.301+0000 7f5d30bcb640 1 -- 172.21.15.28:0/3502908978 >> v2:172.21.15.53:3300/0 conn(0x7f5d2c164990 msgr2=0x7f5d2c15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.301+0000 7f5d30bcb640 1 --2- 172.21.15.28:0/3502908978 >> v2:172.21.15.53:3300/0 conn(0x7f5d2c164990 0x7f5d2c15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.301+0000 7f5d30bcb640 1 -- 172.21.15.28:0/3502908978 >> v1:172.21.15.53:6789/0 conn(0x7f5d2c1606f0 legacy=0x7f5d2c160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.301+0000 7f5d30bcb640 1 -- 172.21.15.28:0/3502908978 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c163eb0 con 0x7f5d2c0b7660 2024-04-25T19:12:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.301+0000 7f5d30bcb640 1 --2- 172.21.15.28:0/3502908978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c0b7a70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c069020 tx=0x7f5d2002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c5d1aed180902 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.301+0000 7f5d2bfff640 1 -- 172.21.15.28:0/3502908978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20037070 con 0x7f5d2c0b7660 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.301+0000 7f5d2bfff640 1 -- 172.21.15.28:0/3502908978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d20032440 con 0x7f5d2c0b7660 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.301+0000 7f5d2bfff640 1 -- 172.21.15.28:0/3502908978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20032810 con 0x7f5d2c0b7660 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.302+0000 7f5d33630640 1 -- 172.21.15.28:0/3502908978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 msgr2=0x7f5d2c0b7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.302+0000 7f5d33630640 1 --2- 172.21.15.28:0/3502908978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c0b7a70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c069020 tx=0x7f5d2002a270 comp rx=0 tx=0).stop 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.302+0000 7f5d33630640 1 -- 172.21.15.28:0/3502908978 shutdown_connections 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.302+0000 7f5d33630640 1 --2- 172.21.15.28:0/3502908978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c0b7a70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.302+0000 7f5d33630640 1 --2- 172.21.15.28:0/3502908978 >> v2:172.21.15.53:3300/0 conn(0x7f5d2c164990 0x7f5d2c15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.302+0000 7f5d33630640 1 -- 172.21.15.28:0/3502908978 >> 172.21.15.28:0/3502908978 conn(0x7f5d2c0d2840 msgr2=0x7f5d2c0d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.302+0000 7f5d33630640 1 -- 172.21.15.28:0/3502908978 shutdown_connections 2024-04-25T19:12:15.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.302+0000 7f5d33630640 1 -- 172.21.15.28:0/3502908978 wait complete. 2024-04-25T19:12:15.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.304+0000 7f5d33630640 1 Processor -- start 2024-04-25T19:12:15.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.304+0000 7f5d33630640 1 -- start start 2024-04-25T19:12:15.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.305+0000 7f5d33630640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c14ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.305+0000 7f5d33630640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d2c1606f0 0x7f5d2c15ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.305+0000 7f5d33630640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c164990 0x7f5d2c15f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.305+0000 7f5d33630640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c0b45d0 con 0x7f5d2c0b7660 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.305+0000 7f5d33630640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2c164470 con 0x7f5d2c164990 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.305+0000 7f5d33630640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2c0b4750 con 0x7f5d2c1606f0 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c14ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c14ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56234/0 (socket says 172.21.15.28:56234) 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 -- 172.21.15.28:0/1829397099 learned_addr learned my addr 172.21.15.28:0/1829397099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d30bcb640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d2c1606f0 0x7f5d2c15ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d31bcd640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c164990 0x7f5d2c15f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 -- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d2c1606f0 msgr2=0x7f5d2c15ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d2c1606f0 0x7f5d2c15ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 -- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c164990 msgr2=0x7f5d2c15f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c164990 0x7f5d2c15f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 -- 172.21.15.28:0/1829397099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2c0cb140 con 0x7f5d2c0b7660 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d31bcd640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c164990 0x7f5d2c15f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:15.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d313cc640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c14ff20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400d600 tx=0x7f5d2400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d29ffb640 1 -- 172.21.15.28:0/1829397099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d240125f0 con 0x7f5d2c0b7660 2024-04-25T19:12:15.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d29ffb640 1 -- 172.21.15.28:0/1829397099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d24019070 con 0x7f5d2c0b7660 2024-04-25T19:12:15.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2c0cb420 con 0x7f5d2c0b7660 2024-04-25T19:12:15.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d29ffb640 1 -- 172.21.15.28:0/1829397099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d240129d0 con 0x7f5d2c0b7660 2024-04-25T19:12:15.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.306+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2c0cb9e0 con 0x7f5d2c0b7660 2024-04-25T19:12:15.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.307+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2c050740 con 0x7f5d2c0b7660 2024-04-25T19:12:15.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.308+0000 7f5d29ffb640 1 -- 172.21.15.28:0/1829397099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5d24012b70 con 0x7f5d2c0b7660 2024-04-25T19:12:15.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.308+0000 7f5d29ffb640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d180744e0 0x7f5d180769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.308+0000 7f5d29ffb640 1 -- 172.21.15.28:0/1829397099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5d24095a70 con 0x7f5d2c0b7660 2024-04-25T19:12:15.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.308+0000 7f5d30bcb640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d180744e0 0x7f5d180769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.309+0000 7f5d30bcb640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d180744e0 0x7f5d180769f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5d20034000 tx=0x7f5d2002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.313+0000 7f5d29ffb640 1 -- 172.21.15.28:0/1829397099 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d2405fe00 con 0x7f5d2c0b7660 2024-04-25T19:12:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.497+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5d2c0d4000 con 0x7f5d2c0b7660 2024-04-25T19:12:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.497+0000 7f5d29ffb640 1 -- 172.21.15.28:0/1829397099 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f5d2405f700 con 0x7f5d2c0b7660 2024-04-25T19:12:15.499 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d180744e0 msgr2=0x7f5d180769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d180744e0 0x7f5d180769f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5d20034000 tx=0x7f5d2002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 msgr2=0x7f5d2c14ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c14ff20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5d2400d600 tx=0x7f5d2400dac0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 shutdown_connections 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5d180744e0 0x7f5d180769f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c164990 0x7f5d2c15f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5d2c1606f0 0x7f5d2c15ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 --2- 172.21.15.28:0/1829397099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d2c0b7660 0x7f5d2c14ff20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.499+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 >> 172.21.15.28:0/1829397099 conn(0x7f5d2c0d2840 msgr2=0x7f5d2c0b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.500+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 shutdown_connections 2024-04-25T19:12:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.500+0000 7f5d33630640 1 -- 172.21.15.28:0/1829397099 wait complete. 2024-04-25T19:12:15.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.625+0000 7fd5774ca640 1 Processor -- start 2024-04-25T19:12:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.625+0000 7fd5774ca640 1 -- start start 2024-04-25T19:12:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd5774ca640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd570164990 0x7fd57015dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd5774ca640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5700b7660 0x7fd5700b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd5774ca640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd570161210 con 0x7fd5701606f0 2024-04-25T19:12:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd5774ca640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fd5700b7fb0 con 0x7fd5700b7660 2024-04-25T19:12:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd5774ca640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd5701614a0 con 0x7fd570164990 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd574a65640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5700b7660 0x7fd5700b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd575266640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd570164990 0x7fd57015dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd575a67640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd5701606f0 0x7fd570160b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:58884/0 (socket says 172.21.15.28:58884) 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd574a65640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5700b7660 0x7fd5700b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:42402/0 (socket says 172.21.15.28:42402) 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd575a67640 1 -- 172.21.15.28:0/3880642194 learned_addr learned my addr 172.21.15.28:0/3880642194 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd575266640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd570164990 0x7fd57015dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:44076/0 (socket says 172.21.15.28:44076) 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd574a65640 1 -- 172.21.15.28:0/3880642194 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd570164990 msgr2=0x7fd57015dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd574a65640 1 --2- 172.21.15.28:0/3880642194 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd570164990 0x7fd57015dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.627+0000 7fd574a65640 1 -- 172.21.15.28:0/3880642194 >> v1:172.21.15.53:6789/0 conn(0x7fd5701606f0 legacy=0x7fd570160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.628+0000 7fd574a65640 1 -- 172.21.15.28:0/3880642194 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd570163eb0 con 0x7fd5700b7660 2024-04-25T19:12:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.628+0000 7fd575266640 1 --2- 172.21.15.28:0/3880642194 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd570164990 0x7fd57015dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.628+0000 7fd574a65640 1 --2- 172.21.15.28:0/3880642194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5700b7660 0x7fd5700b7a70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd5640097a0 tx=0x7fd56402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d52f5ae8db50853 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.628+0000 7fd56ffff640 1 -- 172.21.15.28:0/3880642194 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd564037070 con 0x7fd5700b7660 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd56ffff640 1 -- 172.21.15.28:0/3880642194 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd564032440 con 0x7fd5700b7660 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd56ffff640 1 -- 172.21.15.28:0/3880642194 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd564032810 con 0x7fd5700b7660 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd5774ca640 1 -- 172.21.15.28:0/3880642194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5700b7660 msgr2=0x7fd5700b7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd5774ca640 1 --2- 172.21.15.28:0/3880642194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5700b7660 0x7fd5700b7a70 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd5640097a0 tx=0x7fd56402a180 comp rx=0 tx=0).stop 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd5774ca640 1 -- 172.21.15.28:0/3880642194 shutdown_connections 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd5774ca640 1 --2- 172.21.15.28:0/3880642194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5700b7660 0x7fd5700b7a70 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd5774ca640 1 --2- 172.21.15.28:0/3880642194 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd570164990 0x7fd57015dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd5774ca640 1 -- 172.21.15.28:0/3880642194 >> 172.21.15.28:0/3880642194 conn(0x7fd5700d2840 msgr2=0x7fd5700d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd5774ca640 1 -- 172.21.15.28:0/3880642194 shutdown_connections 2024-04-25T19:12:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.629+0000 7fd5774ca640 1 -- 172.21.15.28:0/3880642194 wait complete. 2024-04-25T19:12:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.631+0000 7fd5774ca640 1 Processor -- start 2024-04-25T19:12:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.631+0000 7fd5774ca640 1 -- start start 2024-04-25T19:12:15.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.632+0000 7fd5774ca640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5700b7660 0x7fd5701e9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.632+0000 7fd5774ca640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5701606f0 0x7fd57014ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.632+0000 7fd5774ca640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd570164990 0x7fd5700cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.632+0000 7fd5774ca640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700b45d0 con 0x7fd5700b7660 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.632+0000 7fd5774ca640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd570164470 con 0x7fd570164990 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.632+0000 7fd5774ca640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd5700b4750 con 0x7fd5701606f0 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5700b7660 0x7fd5701e9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5700b7660 0x7fd5701e9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56238/0 (socket says 172.21.15.28:56238) 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 -- 172.21.15.28:0/2788699401 learned_addr learned my addr 172.21.15.28:0/2788699401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575a67640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd570164990 0x7fd5700cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd574a65640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5701606f0 0x7fd57014ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 -- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5701606f0 msgr2=0x7fd57014ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5701606f0 0x7fd57014ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 -- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd570164990 msgr2=0x7fd5700cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd570164990 0x7fd5700cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 -- 172.21.15.28:0/2788699401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5700cce10 con 0x7fd5700b7660 2024-04-25T19:12:15.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd574a65640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5701606f0 0x7fd57014ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd575266640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5700b7660 0x7fd5701e9b20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd56800e940 tx=0x7fd56800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd56dffb640 1 -- 172.21.15.28:0/2788699401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd568017070 con 0x7fd5700b7660 2024-04-25T19:12:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd56dffb640 1 -- 172.21.15.28:0/2788699401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd56800ca00 con 0x7fd5700b7660 2024-04-25T19:12:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd56dffb640 1 -- 172.21.15.28:0/2788699401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd56800cdd0 con 0x7fd5700b7660 2024-04-25T19:12:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.633+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5701fa3f0 con 0x7fd5700b7660 2024-04-25T19:12:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.634+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5701fa9b0 con 0x7fd5700b7660 2024-04-25T19:12:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.634+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd570050740 con 0x7fd5700b7660 2024-04-25T19:12:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.635+0000 7fd56dffb640 1 -- 172.21.15.28:0/2788699401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd56801a070 con 0x7fd5700b7660 2024-04-25T19:12:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.636+0000 7fd56dffb640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd55c0744b0 0x7fd55c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.636+0000 7fd574a65640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd55c0744b0 0x7fd55c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.636+0000 7fd56dffb640 1 -- 172.21.15.28:0/2788699401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd568095550 con 0x7fd5700b7660 2024-04-25T19:12:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.636+0000 7fd574a65640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd55c0744b0 0x7fd55c0769c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd564034000 tx=0x7fd56402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.640+0000 7fd56dffb640 1 -- 172.21.15.28:0/2788699401 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd56805f8e0 con 0x7fd5700b7660 2024-04-25T19:12:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.827+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5700d4000 con 0x7fd5700b7660 2024-04-25T19:12:15.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.828+0000 7fd56dffb640 1 -- 172.21.15.28:0/2788699401 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fd56805f1e0 con 0x7fd5700b7660 2024-04-25T19:12:15.830 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd55c0744b0 msgr2=0x7fd55c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd55c0744b0 0x7fd55c0769c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd564034000 tx=0x7fd56402e000 comp rx=0 tx=0).stop 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5700b7660 msgr2=0x7fd5701e9b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5700b7660 0x7fd5701e9b20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd56800e940 tx=0x7fd56800ee00 comp rx=0 tx=0).stop 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 shutdown_connections 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd55c0744b0 0x7fd55c0769c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd570164990 0x7fd5700cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd5701606f0 0x7fd57014ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 --2- 172.21.15.28:0/2788699401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5700b7660 0x7fd5701e9b20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 >> 172.21.15.28:0/2788699401 conn(0x7fd5700d2840 msgr2=0x7fd570160b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 shutdown_connections 2024-04-25T19:12:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.830+0000 7fd5774ca640 1 -- 172.21.15.28:0/2788699401 wait complete. 2024-04-25T19:12:15.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.956+0000 7f0e0dc42640 1 Processor -- start 2024-04-25T19:12:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.956+0000 7f0e0dc42640 1 -- start start 2024-04-25T19:12:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.957+0000 7f0e0dc42640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e080b30e0 0x7f0e080cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.957+0000 7f0e0dc42640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e080b3650 con 0x7f0e0809b540 2024-04-25T19:12:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.957+0000 7f0e0dc42640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e0806c710 con 0x7f0e0806d1f0 2024-04-25T19:12:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.957+0000 7f0e0dc42640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0e0806c9f0 con 0x7f0e080b30e0 2024-04-25T19:12:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.957+0000 7f0e06ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0e0809b540 0x7f0e0809b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50570/0 (socket says 172.21.15.28:50570) 2024-04-25T19:12:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.957+0000 7f0e06ffd640 1 -- 172.21.15.28:0/3833656260 learned_addr learned my addr 172.21.15.28:0/3833656260 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e077fe640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e080b30e0 0x7f0e080cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388478755 0 0) 0x7f0e080b3650 con 0x7f0e0809b540 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0df0003520 con 0x7f0e0809b540 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2488868048 0 0) 0x7f0df0003520 con 0x7f0e0809b540 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e080b30e0 msgr2=0x7f0e080cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e080b30e0 0x7f0e080cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 >> v1:172.21.15.53:6789/0 conn(0x7f0e0806d1f0 legacy=0x7f0e0806d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0806ccd0 con 0x7f0e0809b540 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f0dfc001670 con 0x7f0e0809b540 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 >> v1:172.21.15.28:6789/0 conn(0x7f0e0809b540 legacy=0x7f0e0809b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0df00039e0 0x7f0df0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e080b30e0 0x7f0df0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df0004d40 0x7f0df0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0806c9f0 con 0x7f0df0004d40 2024-04-25T19:12:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0df000bb50 con 0x7f0df00039e0 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0df000bd00 con 0x7f0e080b30e0 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e07fff640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df0004d40 0x7f0df0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e077fe640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0df00039e0 0x7f0df0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e06ffd640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e080b30e0 0x7f0df0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e07fff640 1 -- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e080b30e0 msgr2=0x7f0df0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e07fff640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0e080b30e0 0x7f0df0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e07fff640 1 -- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0df00039e0 msgr2=0x7f0df0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.958+0000 7f0e07fff640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0df00039e0 0x7f0df0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.959+0000 7f0e07fff640 1 -- 172.21.15.28:0/3833656260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0df0003520 con 0x7f0df0004d40 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.959+0000 7f0e07fff640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df0004d40 0x7f0df0005150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0e00001830 tx=0x7f0df0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7ea1260712c198 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.959+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00037070 con 0x7f0df0004d40 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.959+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e0002abe0 con 0x7f0df0004d40 2024-04-25T19:12:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.959+0000 7f0e067fc640 1 -- 172.21.15.28:0/3833656260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00001430 con 0x7f0df0004d40 2024-04-25T19:12:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.960+0000 7f0e0dc42640 1 -- 172.21.15.28:0/3833656260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df0004d40 msgr2=0x7f0df0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.960+0000 7f0e0dc42640 1 --2- 172.21.15.28:0/3833656260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0df0004d40 0x7f0df0005150 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0e00001830 tx=0x7f0df0005830 comp rx=0 tx=0).stop 2024-04-25T19:12:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.960+0000 7f0e07fff640 1 -- 172.21.15.28:0/3833656260 reap_dead start 2024-04-25T19:12:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.960+0000 7f0e0dc42640 1 -- 172.21.15.28:0/3833656260 shutdown_connections 2024-04-25T19:12:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.960+0000 7f0e0dc42640 1 -- 172.21.15.28:0/3833656260 >> 172.21.15.28:0/3833656260 conn(0x7f0e080b7620 msgr2=0x7f0e080b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.960+0000 7f0e0dc42640 1 -- 172.21.15.28:0/3833656260 shutdown_connections 2024-04-25T19:12:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.960+0000 7f0e0dc42640 1 -- 172.21.15.28:0/3833656260 wait complete. 2024-04-25T19:12:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.962+0000 7f0e0dc42640 1 Processor -- start 2024-04-25T19:12:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.962+0000 7f0e0dc42640 1 -- start start 2024-04-25T19:12:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.963+0000 7f0e0dc42640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0df0004d40 0x7f0e0815b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.963+0000 7f0e0dc42640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e081f8390 0x7f0e0815fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.963+0000 7f0e0dc42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e081f7710 0x7f0e081f7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.963+0000 7f0e0dc42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0e080cf950 con 0x7f0e081f7710 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.963+0000 7f0e0dc42640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0e080cf7d0 con 0x7f0e081f8390 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.963+0000 7f0e0dc42640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0e080cfad0 con 0x7f0df0004d40 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e081f7710 0x7f0e081f7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e081f7710 0x7f0e081f7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56262/0 (socket says 172.21.15.28:56262) 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 -- 172.21.15.28:0/2551696424 learned_addr learned my addr 172.21.15.28:0/2551696424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e06ffd640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e081f8390 0x7f0e0815fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e077fe640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0df0004d40 0x7f0e0815b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 -- 172.21.15.28:0/2551696424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0df0004d40 msgr2=0x7f0e0815b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0df0004d40 0x7f0e0815b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 -- 172.21.15.28:0/2551696424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e081f8390 msgr2=0x7f0e0815fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e081f8390 0x7f0e0815fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 -- 172.21.15.28:0/2551696424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e081f8060 con 0x7f0e081f7710 2024-04-25T19:12:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e07fff640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e081f7710 0x7f0e081f7b20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0e00001950 tx=0x7f0e00001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0deffff640 1 -- 172.21.15.28:0/2551696424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00037040 con 0x7f0e081f7710 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0deffff640 1 -- 172.21.15.28:0/2551696424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e00041070 con 0x7f0e081f7710 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0deffff640 1 -- 172.21.15.28:0/2551696424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00001430 con 0x7f0e081f7710 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0815c950 con 0x7f0e081f7710 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.964+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0815bb90 con 0x7f0e081f7710 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.965+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e08050740 con 0x7f0e081f7710 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.966+0000 7f0deffff640 1 -- 172.21.15.28:0/2551696424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0e0003a070 con 0x7f0e081f7710 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.966+0000 7f0deffff640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0df40744b0 0x7f0df40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.967+0000 7f0deffff640 1 -- 172.21.15.28:0/2551696424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0e000b5060 con 0x7f0e081f7710 2024-04-25T19:12:15.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.967+0000 7f0e077fe640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0df40744b0 0x7f0df40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:15.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.967+0000 7f0e077fe640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0df40744b0 0x7f0df40769c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0e080b1f90 tx=0x7f0df800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:15.971+0000 7f0deffff640 1 -- 172.21.15.28:0/2551696424 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e081f8060 con 0x7f0e081f7710 2024-04-25T19:12:16.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.156+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0e080cf5b0 con 0x7f0e081f7710 2024-04-25T19:12:16.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.156+0000 7f0deffff640 1 -- 172.21.15.28:0/2551696424 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f0e00005470 con 0x7f0e081f7710 2024-04-25T19:12:16.158 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:16.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0df40744b0 msgr2=0x7f0df40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0df40744b0 0x7f0df40769c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0e080b1f90 tx=0x7f0df800b040 comp rx=0 tx=0).stop 2024-04-25T19:12:16.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e081f7710 msgr2=0x7f0e081f7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e081f7710 0x7f0e081f7b20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0e00001950 tx=0x7f0e00001230 comp rx=0 tx=0).stop 2024-04-25T19:12:16.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 shutdown_connections 2024-04-25T19:12:16.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0df40744b0 0x7f0df40769c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e081f7710 0x7f0e081f7b20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e081f8390 0x7f0e0815fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 --2- 172.21.15.28:0/2551696424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0df0004d40 0x7f0e0815b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 >> 172.21.15.28:0/2551696424 conn(0x7f0e080b7620 msgr2=0x7f0e08071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:16.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 shutdown_connections 2024-04-25T19:12:16.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.158+0000 7f0e0dc42640 1 -- 172.21.15.28:0/2551696424 wait complete. 2024-04-25T19:12:16.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:16.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.284+0000 7f4e20242640 1 Processor -- start 2024-04-25T19:12:16.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.284+0000 7f4e20242640 1 -- start start 2024-04-25T19:12:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e20242640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e18160310 0x7f4e180d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e20242640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e18160720 con 0x7f4e180b9920 2024-04-25T19:12:16.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e20242640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e180d5640 con 0x7f4e180b79f0 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e20242640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f4e180d0f40 con 0x7f4e18160310 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e1dfde640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e18160310 0x7f4e180d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e1dfde640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e18160310 0x7f4e180d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56268/0 (socket says 172.21.15.28:56268) 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e1dfde640 1 -- 172.21.15.28:0/2750862218 learned_addr learned my addr 172.21.15.28:0/2750862218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e1e7df640 1 --1- 172.21.15.28:0/2750862218 >> v1:172.21.15.28:6789/0 conn(0x7f4e180b9920 0x7f4e180b9d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50582/0 (socket says 172.21.15.28:50582) 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e1dfde640 1 -- 172.21.15.28:0/2750862218 >> v1:172.21.15.53:6789/0 conn(0x7f4e180b79f0 legacy=0x7f4e180b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e1dfde640 1 -- 172.21.15.28:0/2750862218 >> v1:172.21.15.28:6789/0 conn(0x7f4e180b9920 legacy=0x7f4e180b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e1dfde640 1 -- 172.21.15.28:0/2750862218 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e180d1220 con 0x7f4e18160310 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.286+0000 7f4e1cfdc640 1 -- 172.21.15.28:0/2750862218 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 893462324 0 0) 0x7f4e18160720 con 0x7f4e180b9920 2024-04-25T19:12:16.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.287+0000 7f4e1dfde640 1 --2- 172.21.15.28:0/2750862218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e18160310 0x7f4e180d5100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009870 tx=0x7f4e0c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a1ea1df19dbcbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.287+0000 7f4e1cfdc640 1 -- 172.21.15.28:0/2750862218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c037070 con 0x7f4e18160310 2024-04-25T19:12:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.287+0000 7f4e1cfdc640 1 -- 172.21.15.28:0/2750862218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e0c032440 con 0x7f4e18160310 2024-04-25T19:12:16.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.287+0000 7f4e1cfdc640 1 -- 172.21.15.28:0/2750862218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c032810 con 0x7f4e18160310 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.288+0000 7f4e20242640 1 -- 172.21.15.28:0/2750862218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e18160310 msgr2=0x7f4e180d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.288+0000 7f4e20242640 1 --2- 172.21.15.28:0/2750862218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e18160310 0x7f4e180d5100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009870 tx=0x7f4e0c02a2a0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.288+0000 7f4e20242640 1 -- 172.21.15.28:0/2750862218 shutdown_connections 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.288+0000 7f4e20242640 1 --2- 172.21.15.28:0/2750862218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e18160310 0x7f4e180d5100 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.288+0000 7f4e20242640 1 -- 172.21.15.28:0/2750862218 >> 172.21.15.28:0/2750862218 conn(0x7f4e180d2a50 msgr2=0x7f4e180d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.288+0000 7f4e20242640 1 -- 172.21.15.28:0/2750862218 shutdown_connections 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.288+0000 7f4e20242640 1 -- 172.21.15.28:0/2750862218 wait complete. 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.290+0000 7f4e20242640 1 Processor -- start 2024-04-25T19:12:16.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.290+0000 7f4e20242640 1 -- start start 2024-04-25T19:12:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e20242640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e180b79f0 0x7f4e1814be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e20242640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e180b9920 0x7f4e180c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e20242640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18160310 0x7f4e181eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e20242640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e180d0bf0 con 0x7f4e180b79f0 2024-04-25T19:12:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e20242640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4e180b9400 con 0x7f4e18160310 2024-04-25T19:12:16.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e20242640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4e180cc950 con 0x7f4e180b9920 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1dfde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e180b79f0 0x7f4e1814be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1dfde640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e180b79f0 0x7f4e1814be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56272/0 (socket says 172.21.15.28:56272) 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1dfde640 1 -- 172.21.15.28:0/503316884 learned_addr learned my addr 172.21.15.28:0/503316884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1d7dd640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e180b9920 0x7f4e180c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1e7df640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18160310 0x7f4e181eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1dfde640 1 -- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e180b9920 msgr2=0x7f4e180c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1dfde640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e180b9920 0x7f4e180c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1dfde640 1 -- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18160310 msgr2=0x7f4e181eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1dfde640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18160310 0x7f4e181eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1dfde640 1 -- 172.21.15.28:0/503316884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e180ca410 con 0x7f4e180b79f0 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1e7df640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18160310 0x7f4e181eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.291+0000 7f4e1d7dd640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e180b9920 0x7f4e180c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.292+0000 7f4e1dfde640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e180b79f0 0x7f4e1814be90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009870 tx=0x7f4e0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.292+0000 7f4e0a7fc640 1 -- 172.21.15.28:0/503316884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c037040 con 0x7f4e180b79f0 2024-04-25T19:12:16.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.292+0000 7f4e0a7fc640 1 -- 172.21.15.28:0/503316884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e0c004250 con 0x7f4e180b79f0 2024-04-25T19:12:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.292+0000 7f4e0a7fc640 1 -- 172.21.15.28:0/503316884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c03c670 con 0x7f4e180b79f0 2024-04-25T19:12:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.292+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e180ca6f0 con 0x7f4e180b79f0 2024-04-25T19:12:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.292+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e180cacb0 con 0x7f4e180b79f0 2024-04-25T19:12:16.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.293+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e18050740 con 0x7f4e180b79f0 2024-04-25T19:12:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.294+0000 7f4e0a7fc640 1 -- 172.21.15.28:0/503316884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4e0c03a070 con 0x7f4e180b79f0 2024-04-25T19:12:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.294+0000 7f4e0a7fc640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dfc0744b0 0x7f4dfc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.294+0000 7f4e1d7dd640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dfc0744b0 0x7f4dfc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.294+0000 7f4e0a7fc640 1 -- 172.21.15.28:0/503316884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4e0c0b5f40 con 0x7f4e180b79f0 2024-04-25T19:12:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.295+0000 7f4e1d7dd640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dfc0744b0 0x7f4dfc0769c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4e180d1c30 tx=0x7f4e14009570 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:16.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.299+0000 7f4e0a7fc640 1 -- 172.21.15.28:0/503316884 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e0c041020 con 0x7f4e180b79f0 2024-04-25T19:12:16.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.483+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4e180c9280 con 0x7f4e180b79f0 2024-04-25T19:12:16.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.484+0000 7f4e0a7fc640 1 -- 172.21.15.28:0/503316884 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f4e0c080360 con 0x7f4e180b79f0 2024-04-25T19:12:16.486 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dfc0744b0 msgr2=0x7f4dfc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dfc0744b0 0x7f4dfc0769c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4e180d1c30 tx=0x7f4e14009570 comp rx=0 tx=0).stop 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e180b79f0 msgr2=0x7f4e1814be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e180b79f0 0x7f4e1814be90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c009870 tx=0x7f4e0c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 shutdown_connections 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4dfc0744b0 0x7f4dfc0769c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18160310 0x7f4e181eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4e180b9920 0x7f4e180c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 --2- 172.21.15.28:0/503316884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e180b79f0 0x7f4e1814be90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 >> 172.21.15.28:0/503316884 conn(0x7f4e180d2a50 msgr2=0x7f4e1815f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 shutdown_connections 2024-04-25T19:12:16.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.486+0000 7f4e20242640 1 -- 172.21.15.28:0/503316884 wait complete. 2024-04-25T19:12:16.540 INFO:tasks.cephfs_test_runner:Starting test: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2024-04-25T19:12:16.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.651+0000 7ffbf4c5a640 1 Processor -- start 2024-04-25T19:12:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.651+0000 7ffbf4c5a640 1 -- start start 2024-04-25T19:12:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbf4c5a640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffbf006e5e0 0x7ffbf0161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbf4c5a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf015fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbf4c5a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbf0163e80 con 0x7ffbf00b5130 2024-04-25T19:12:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbf4c5a640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7ffbf0163c30 con 0x7ffbf006e5e0 2024-04-25T19:12:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbf4c5a640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ffbf0164a90 con 0x7ffbf015f7d0 2024-04-25T19:12:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbedd9b640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffbf006e5e0 0x7ffbf0161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbeed9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf015fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbee59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ffbf00b5130 0x7ffbf006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:58914/0 (socket says 172.21.15.28:58914) 2024-04-25T19:12:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.653+0000 7ffbee59c640 1 -- 172.21.15.28:0/1481913349 learned_addr learned my addr 172.21.15.28:0/1481913349 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.654+0000 7ffbed59a640 1 -- 172.21.15.28:0/1481913349 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265394457 0 0) 0x7ffbf0163e80 con 0x7ffbf00b5130 2024-04-25T19:12:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.654+0000 7ffbed59a640 1 -- 172.21.15.28:0/1481913349 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffbdc003520 con 0x7ffbf00b5130 2024-04-25T19:12:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.654+0000 7ffbeed9d640 1 -- 172.21.15.28:0/1481913349 >> v2:172.21.15.53:3300/0 conn(0x7ffbf006e5e0 msgr2=0x7ffbf0161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.654+0000 7ffbeed9d640 1 --2- 172.21.15.28:0/1481913349 >> v2:172.21.15.53:3300/0 conn(0x7ffbf006e5e0 0x7ffbf0161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.654+0000 7ffbeed9d640 1 -- 172.21.15.28:0/1481913349 >> v1:172.21.15.53:6789/0 conn(0x7ffbf00b5130 legacy=0x7ffbf006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.654+0000 7ffbeed9d640 1 -- 172.21.15.28:0/1481913349 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbf0164160 con 0x7ffbf015f7d0 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.654+0000 7ffbedd9b640 1 --2- 172.21.15.28:0/1481913349 >> v2:172.21.15.53:3300/0 conn(0x7ffbf006e5e0 0x7ffbf0161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.655+0000 7ffbeed9d640 1 --2- 172.21.15.28:0/1481913349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf015fbe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffbe80097a0 tx=0x7ffbe802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24da0685b85cdc30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.655+0000 7ffbed59a640 1 -- 172.21.15.28:0/1481913349 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe8037070 con 0x7ffbf015f7d0 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.655+0000 7ffbed59a640 1 -- 172.21.15.28:0/1481913349 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbe8032440 con 0x7ffbf015f7d0 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.655+0000 7ffbed59a640 1 -- 172.21.15.28:0/1481913349 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe80327f0 con 0x7ffbf015f7d0 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.655+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/1481913349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 msgr2=0x7ffbf015fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.655+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/1481913349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf015fbe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffbe80097a0 tx=0x7ffbe802a180 comp rx=0 tx=0).stop 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.656+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/1481913349 shutdown_connections 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.656+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/1481913349 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf015fbe0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.656+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/1481913349 >> v2:172.21.15.53:3300/0 conn(0x7ffbf006e5e0 0x7ffbf0161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.656+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/1481913349 >> 172.21.15.28:0/1481913349 conn(0x7ffbf00b7ad0 msgr2=0x7ffbf00b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.656+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/1481913349 shutdown_connections 2024-04-25T19:12:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.656+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/1481913349 wait complete. 2024-04-25T19:12:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.658+0000 7ffbf4c5a640 1 Processor -- start 2024-04-25T19:12:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.658+0000 7ffbf4c5a640 1 -- start start 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbf4c5a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf00b5130 0x7ffbf015ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbf4c5a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf01569a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbf4c5a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbf01f6220 0x7ffbf01f6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbf4c5a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf0160120 con 0x7ffbf00b5130 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbf4c5a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ffbf0164720 con 0x7ffbf01f6220 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbf4c5a640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ffbf01602a0 con 0x7ffbf015f7d0 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbee59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf00b5130 0x7ffbf015ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbee59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf00b5130 0x7ffbf015ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56288/0 (socket says 172.21.15.28:56288) 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbee59c640 1 -- 172.21.15.28:0/434190894 learned_addr learned my addr 172.21.15.28:0/434190894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbeed9d640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbf01f6220 0x7ffbf01f6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbedd9b640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf01569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbee59c640 1 -- 172.21.15.28:0/434190894 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 msgr2=0x7ffbf01569a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbee59c640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf01569a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbee59c640 1 -- 172.21.15.28:0/434190894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbf01f6220 msgr2=0x7ffbf01f6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbee59c640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbf01f6220 0x7ffbf01f6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.659+0000 7ffbee59c640 1 -- 172.21.15.28:0/434190894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbf00ba210 con 0x7ffbf00b5130 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.660+0000 7ffbee59c640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf00b5130 0x7ffbf015ab00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffbe000ca50 tx=0x7ffbe000cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.660+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/434190894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0017070 con 0x7ffbf00b5130 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.660+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/434190894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbe000aaf0 con 0x7ffbf00b5130 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.660+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbf00ba4f0 con 0x7ffbf00b5130 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.660+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/434190894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbe0012400 con 0x7ffbf00b5130 2024-04-25T19:12:16.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.660+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbf01482f0 con 0x7ffbf00b5130 2024-04-25T19:12:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.661+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbf0148550 con 0x7ffbf00b5130 2024-04-25T19:12:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.661+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/434190894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffbe001a070 con 0x7ffbf00b5130 2024-04-25T19:12:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.662+0000 7ffbe6ffd640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ffbd00744e0 0x7ffbd00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.662+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/434190894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ffbe0095300 con 0x7ffbf00b5130 2024-04-25T19:12:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.662+0000 7ffbedd9b640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ffbd00744e0 0x7ffbd00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.663+0000 7ffbedd9b640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ffbd00744e0 0x7ffbd00769f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffbf00d1650 tx=0x7ffbd8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:16.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.667+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/434190894 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffbe005f690 con 0x7ffbf00b5130 2024-04-25T19:12:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.858+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffbf0050740 con 0x7ffbf00b5130 2024-04-25T19:12:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.859+0000 7ffbe6ffd640 1 -- 172.21.15.28:0/434190894 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7ffbe005ef90 con 0x7ffbf00b5130 2024-04-25T19:12:16.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:16.861 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ffbd00744e0 msgr2=0x7ffbd00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ffbd00744e0 0x7ffbd00769f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ffbf00d1650 tx=0x7ffbd8008040 comp rx=0 tx=0).stop 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf00b5130 msgr2=0x7ffbf015ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf00b5130 0x7ffbf015ab00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ffbe000ca50 tx=0x7ffbe000cf10 comp rx=0 tx=0).stop 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 shutdown_connections 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ffbd00744e0 0x7ffbd00769f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbf01f6220 0x7ffbf01f6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ffbf015f7d0 0x7ffbf01569a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 --2- 172.21.15.28:0/434190894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbf00b5130 0x7ffbf015ab00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.861+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 >> 172.21.15.28:0/434190894 conn(0x7ffbf00b7ad0 msgr2=0x7ffbf015ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.862+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 shutdown_connections 2024-04-25T19:12:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.862+0000 7ffbf4c5a640 1 -- 172.21.15.28:0/434190894 wait complete. 2024-04-25T19:12:16.878 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:16.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.992+0000 7f0d30292640 1 Processor -- start 2024-04-25T19:12:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.992+0000 7f0d30292640 1 -- start start 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d30292640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d28160220 0x7f0d28164800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d30292640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d280d15b0 con 0x7f0d280b4c80 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d30292640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d280d12d0 con 0x7f0d280b7fd0 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d30292640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f0d28163d20 con 0x7f0d28160220 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d2e82f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d28160220 0x7f0d28164800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d2e02e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0d280b4c80 0x7f0d280b78c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50594/0 (socket says 172.21.15.28:50594) 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d2e02e640 1 -- 172.21.15.28:0/4268988510 learned_addr learned my addr 172.21.15.28:0/4268988510 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d2e82f640 1 -- 172.21.15.28:0/4268988510 >> v1:172.21.15.53:6789/0 conn(0x7f0d280b7fd0 legacy=0x7f0d280d4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d2e82f640 1 -- 172.21.15.28:0/4268988510 >> v1:172.21.15.28:6789/0 conn(0x7f0d280b4c80 legacy=0x7f0d280b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.994+0000 7f0d2e82f640 1 -- 172.21.15.28:0/4268988510 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d28164000 con 0x7f0d28160220 2024-04-25T19:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.995+0000 7f0d2e82f640 1 --2- 172.21.15.28:0/4268988510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d28164800 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009d90 tx=0x7f0d1802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9044d3c8ec80a24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.995+0000 7f0d2d02c640 1 -- 172.21.15.28:0/4268988510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18007740 con 0x7f0d28160220 2024-04-25T19:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.995+0000 7f0d2d02c640 1 -- 172.21.15.28:0/4268988510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d180078e0 con 0x7f0d28160220 2024-04-25T19:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.995+0000 7f0d2d02c640 1 -- 172.21.15.28:0/4268988510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18007cb0 con 0x7f0d28160220 2024-04-25T19:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.996+0000 7f0d30292640 1 -- 172.21.15.28:0/4268988510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 msgr2=0x7f0d28164800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.996+0000 7f0d30292640 1 --2- 172.21.15.28:0/4268988510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d28164800 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f0d18009d90 tx=0x7f0d1802a6a0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.996+0000 7f0d30292640 1 -- 172.21.15.28:0/4268988510 shutdown_connections 2024-04-25T19:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.996+0000 7f0d30292640 1 --2- 172.21.15.28:0/4268988510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d28164800 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.996+0000 7f0d30292640 1 -- 172.21.15.28:0/4268988510 >> 172.21.15.28:0/4268988510 conn(0x7f0d280d2980 msgr2=0x7f0d280d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.996+0000 7f0d30292640 1 -- 172.21.15.28:0/4268988510 shutdown_connections 2024-04-25T19:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.996+0000 7f0d30292640 1 -- 172.21.15.28:0/4268988510 wait complete. 2024-04-25T19:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.997+0000 7f0d30292640 1 Processor -- start 2024-04-25T19:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.998+0000 7f0d30292640 1 -- start start 2024-04-25T19:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d30292640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d280b4c80 0x7f0d280ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d30292640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d281eac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d30292640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d28159b70 0x7f0d280ba0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d30292640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d280b53c0 con 0x7f0d28160220 2024-04-25T19:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d30292640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0d280b5240 con 0x7f0d280b4c80 2024-04-25T19:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d30292640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0d280b46c0 con 0x7f0d28159b70 2024-04-25T19:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d281eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d281eac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56308/0 (socket says 172.21.15.28:56308) 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 -- 172.21.15.28:0/502052230 learned_addr learned my addr 172.21.15.28:0/502052230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2e02e640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d280b4c80 0x7f0d280ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2e82f640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d28159b70 0x7f0d280ba0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 -- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d28159b70 msgr2=0x7f0d280ba0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d28159b70 0x7f0d280ba0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 -- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d280b4c80 msgr2=0x7f0d280ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d280b4c80 0x7f0d280ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 -- 172.21.15.28:0/502052230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d280cf9b0 con 0x7f0d28160220 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2e02e640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d280b4c80 0x7f0d280ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2e82f640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d28159b70 0x7f0d280ba0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:16.999+0000 7f0d2d82d640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d281eac90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00b410 tx=0x7f0d1c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.000+0000 7f0d26ffd640 1 -- 172.21.15.28:0/502052230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c015020 con 0x7f0d28160220 2024-04-25T19:12:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.000+0000 7f0d26ffd640 1 -- 172.21.15.28:0/502052230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d1c011830 con 0x7f0d28160220 2024-04-25T19:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.000+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d280cf110 con 0x7f0d28160220 2024-04-25T19:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.000+0000 7f0d26ffd640 1 -- 172.21.15.28:0/502052230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1c011c20 con 0x7f0d28160220 2024-04-25T19:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.000+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d280cf6d0 con 0x7f0d28160220 2024-04-25T19:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.001+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d28050740 con 0x7f0d28160220 2024-04-25T19:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.001+0000 7f0d26ffd640 1 -- 172.21.15.28:0/502052230 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0d1c018020 con 0x7f0d28160220 2024-04-25T19:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.002+0000 7f0d26ffd640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d140744b0 0x7f0d140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.002+0000 7f0d2e02e640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d140744b0 0x7f0d140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.002+0000 7f0d26ffd640 1 -- 172.21.15.28:0/502052230 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0d1c0952e0 con 0x7f0d28160220 2024-04-25T19:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.002+0000 7f0d2e02e640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d140744b0 0x7f0d140769c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0d20000f60 tx=0x7f0d20009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.007+0000 7f0d26ffd640 1 -- 172.21.15.28:0/502052230 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d1c05f790 con 0x7f0d28160220 2024-04-25T19:12:17.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.194+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0d280d4730 con 0x7f0d28160220 2024-04-25T19:12:17.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.194+0000 7f0d26ffd640 1 -- 172.21.15.28:0/502052230 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f0d1c05f090 con 0x7f0d28160220 2024-04-25T19:12:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d140744b0 msgr2=0x7f0d140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d140744b0 0x7f0d140769c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0d20000f60 tx=0x7f0d20009040 comp rx=0 tx=0).stop 2024-04-25T19:12:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 msgr2=0x7f0d281eac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d281eac90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0d1c00b410 tx=0x7f0d1c00b8d0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 shutdown_connections 2024-04-25T19:12:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d140744b0 0x7f0d140769c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d28159b70 0x7f0d280ba0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d28160220 0x7f0d281eac90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 --2- 172.21.15.28:0/502052230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d280b4c80 0x7f0d280ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 >> 172.21.15.28:0/502052230 conn(0x7f0d280d2980 msgr2=0x7f0d2815a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:17.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 shutdown_connections 2024-04-25T19:12:17.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.196+0000 7f0d30292640 1 -- 172.21.15.28:0/502052230 wait complete. 2024-04-25T19:12:17.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:17.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.323+0000 7f9665451640 1 Processor -- start 2024-04-25T19:12:17.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.323+0000 7f9665451640 1 -- start start 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f9665451640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f96600b7d70 0x7f966015dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f9665451640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9660161210 con 0x7f96601606f0 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f9665451640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96601614a0 con 0x7f9660164990 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f9665451640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f9660163eb0 con 0x7f96600b7d70 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f965f7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f96601606f0 0x7f9660160b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50604/0 (socket says 172.21.15.28:50604) 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f965f7fe640 1 -- 172.21.15.28:0/1863986103 learned_addr learned my addr 172.21.15.28:0/1863986103 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2009741502 0 0) 0x7f9660161210 con 0x7f96601606f0 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f965e7fc640 1 --2- 172.21.15.28:0/1863986103 >> v2:172.21.15.53:3300/0 conn(0x7f96600b7d70 0x7f966015dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.324+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f964c003520 con 0x7f96601606f0 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 62026271 0 0) 0x7f964c003520 con 0x7f96601606f0 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 >> v2:172.21.15.53:3300/0 conn(0x7f96600b7d70 msgr2=0x7f966015dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 --2- 172.21.15.28:0/1863986103 >> v2:172.21.15.53:3300/0 conn(0x7f96600b7d70 0x7f966015dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 >> v1:172.21.15.53:6789/0 conn(0x7f9660164990 legacy=0x7f96600b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9660164190 con 0x7f96601606f0 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f9658001670 con 0x7f96601606f0 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 >> v1:172.21.15.28:6789/0 conn(0x7f96601606f0 legacy=0x7f9660160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c0039b0 0x7f964c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964c004800 0x7f964c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c0051c0 0x7f964c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9660163eb0 con 0x7f964c004800 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f964c00c160 con 0x7f964c0039b0 2024-04-25T19:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f964c00c360 con 0x7f964c0051c0 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965e7fc640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964c004800 0x7f964c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965effd640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c0039b0 0x7f964c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965f7fe640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c0051c0 0x7f964c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965e7fc640 1 -- 172.21.15.28:0/1863986103 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c0051c0 msgr2=0x7f964c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965e7fc640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c0051c0 0x7f964c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965e7fc640 1 -- 172.21.15.28:0/1863986103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c0039b0 msgr2=0x7f964c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965e7fc640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f964c0039b0 0x7f964c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965e7fc640 1 -- 172.21.15.28:0/1863986103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f964c003520 con 0x7f964c004800 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.325+0000 7f965f7fe640 1 -- 172.21.15.28:0/1863986103 reap_dead start 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.326+0000 7f965e7fc640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964c004800 0x7f964c004c10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9648009ac0 tx=0x7f964c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab5c371fb8e9479 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.326+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648036070 con 0x7f964c004800 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.326+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9648031400 con 0x7f964c004800 2024-04-25T19:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.326+0000 7f965dffb640 1 -- 172.21.15.28:0/1863986103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648009690 con 0x7f964c004800 2024-04-25T19:12:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.327+0000 7f9665451640 1 -- 172.21.15.28:0/1863986103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964c004800 msgr2=0x7f964c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.327+0000 7f9665451640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964c004800 0x7f964c004c10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9648009ac0 tx=0x7f964c005830 comp rx=0 tx=0).stop 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.327+0000 7f9665451640 1 -- 172.21.15.28:0/1863986103 shutdown_connections 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.327+0000 7f9665451640 1 --2- 172.21.15.28:0/1863986103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f964c004800 0x7f964c004c10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.327+0000 7f9665451640 1 -- 172.21.15.28:0/1863986103 >> 172.21.15.28:0/1863986103 conn(0x7f96600d2840 msgr2=0x7f96600d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.327+0000 7f9665451640 1 -- 172.21.15.28:0/1863986103 shutdown_connections 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.327+0000 7f9665451640 1 -- 172.21.15.28:0/1863986103 wait complete. 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.329+0000 7f9665451640 1 Processor -- start 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.329+0000 7f9665451640 1 -- start start 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.330+0000 7f9665451640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c004800 0x7f966014e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.330+0000 7f9665451640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96600ccf30 0x7f96600cd340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.330+0000 7f9665451640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96601627d0 0x7f9660162be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.330+0000 7f9665451640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f966015de30 con 0x7f96600ccf30 2024-04-25T19:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.330+0000 7f9665451640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f966015dcb0 con 0x7f96601627d0 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.330+0000 7f9665451640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f966015dfb0 con 0x7f964c004800 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.330+0000 7f965e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96600ccf30 0x7f96600cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.330+0000 7f965e7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96600ccf30 0x7f96600cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56328/0 (socket says 172.21.15.28:56328) 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965e7fc640 1 -- 172.21.15.28:0/2601126230 learned_addr learned my addr 172.21.15.28:0/2601126230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965effd640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c004800 0x7f966014e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965f7fe640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96601627d0 0x7f9660162be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965e7fc640 1 -- 172.21.15.28:0/2601126230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c004800 msgr2=0x7f966014e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965e7fc640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c004800 0x7f966014e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965e7fc640 1 -- 172.21.15.28:0/2601126230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96601627d0 msgr2=0x7f9660162be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965e7fc640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96601627d0 0x7f9660162be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965e7fc640 1 -- 172.21.15.28:0/2601126230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9660163120 con 0x7f96600ccf30 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f965e7fc640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96600ccf30 0x7f96600cd340 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9648034040 tx=0x7f9648004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f96577fe640 1 -- 172.21.15.28:0/2601126230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964803f020 con 0x7f96600ccf30 2024-04-25T19:12:17.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f96577fe640 1 -- 172.21.15.28:0/2601126230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9648037040 con 0x7f96600ccf30 2024-04-25T19:12:17.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f966014e640 con 0x7f96600ccf30 2024-04-25T19:12:17.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f96577fe640 1 -- 172.21.15.28:0/2601126230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9648036070 con 0x7f96600ccf30 2024-04-25T19:12:17.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.331+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f966014f1c0 con 0x7f96600ccf30 2024-04-25T19:12:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.332+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9660050740 con 0x7f96600ccf30 2024-04-25T19:12:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.333+0000 7f96577fe640 1 -- 172.21.15.28:0/2601126230 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9648042020 con 0x7f96600ccf30 2024-04-25T19:12:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.334+0000 7f96577fe640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f96400744b0 0x7f96400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.334+0000 7f965effd640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f96400744b0 0x7f96400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.334+0000 7f96577fe640 1 -- 172.21.15.28:0/2601126230 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f96480b5720 con 0x7f96600ccf30 2024-04-25T19:12:17.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.334+0000 7f965effd640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f96400744b0 0x7f96400769c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9650000f80 tx=0x7f9650009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.338+0000 7f96577fe640 1 -- 172.21.15.28:0/2601126230 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9648049030 con 0x7f96600ccf30 2024-04-25T19:12:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.522+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f96600d4000 con 0x7f96600ccf30 2024-04-25T19:12:17.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.523+0000 7f96577fe640 1 -- 172.21.15.28:0/2601126230 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f9660163120 con 0x7f96600ccf30 2024-04-25T19:12:17.525 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.524+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f96400744b0 msgr2=0x7f96400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.524+0000 7f9665451640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f96400744b0 0x7f96400769c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9650000f80 tx=0x7f9650009040 comp rx=0 tx=0).stop 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.524+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96600ccf30 msgr2=0x7f96600cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.524+0000 7f9665451640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96600ccf30 0x7f96600cd340 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9648034040 tx=0x7f9648004730 comp rx=0 tx=0).stop 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.525+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 shutdown_connections 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.525+0000 7f9665451640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f96400744b0 0x7f96400769c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.525+0000 7f9665451640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96601627d0 0x7f9660162be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.525+0000 7f9665451640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96600ccf30 0x7f96600cd340 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.525+0000 7f9665451640 1 --2- 172.21.15.28:0/2601126230 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f964c004800 0x7f966014e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.525+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 >> 172.21.15.28:0/2601126230 conn(0x7f96600d2840 msgr2=0x7f96600b66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.525+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 shutdown_connections 2024-04-25T19:12:17.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.525+0000 7f9665451640 1 -- 172.21.15.28:0/2601126230 wait complete. 2024-04-25T19:12:17.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.651+0000 7f5006dd5640 1 Processor -- start 2024-04-25T19:12:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.651+0000 7f5006dd5640 1 -- start start 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.652+0000 7f5006dd5640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50000d0ee0 0x7f50000affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.652+0000 7f5006dd5640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f500006f5b0 con 0x7f500015aaa0 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.652+0000 7f5006dd5640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50000d14a0 con 0x7f50000b5020 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.652+0000 7f5006dd5640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f50000af290 con 0x7f50000d0ee0 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.653+0000 7f4ff7fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50000d0ee0 0x7f50000affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.653+0000 7f5004b71640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f500015aaa0 0x7f50000b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50620/0 (socket says 172.21.15.28:50620) 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.653+0000 7f4ff7fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50000d0ee0 0x7f50000affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56330/0 (socket says 172.21.15.28:56330) 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.653+0000 7f5004b71640 1 -- 172.21.15.28:0/1555737073 learned_addr learned my addr 172.21.15.28:0/1555737073 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.653+0000 7f4ff7fff640 1 -- 172.21.15.28:0/1555737073 >> v1:172.21.15.53:6789/0 conn(0x7f50000b5020 legacy=0x7f5000160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.653+0000 7f4ff7fff640 1 -- 172.21.15.28:0/1555737073 >> v1:172.21.15.28:6789/0 conn(0x7f500015aaa0 legacy=0x7f50000b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.653+0000 7f4ff7fff640 1 -- 172.21.15.28:0/1555737073 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50000af4e0 con 0x7f50000d0ee0 2024-04-25T19:12:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.653+0000 7f4ff77fe640 1 -- 172.21.15.28:0/1555737073 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279421494 0 0) 0x7f500006f5b0 con 0x7f500015aaa0 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f4ff7fff640 1 --2- 172.21.15.28:0/1555737073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000d0ee0 0x7f50000affc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f50000af410 tx=0x7f4ff002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a019a3d568be88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f4ff77fe640 1 -- 172.21.15.28:0/1555737073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0037070 con 0x7f50000d0ee0 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f4ff77fe640 1 -- 172.21.15.28:0/1555737073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff0032440 con 0x7f50000d0ee0 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f4ff77fe640 1 -- 172.21.15.28:0/1555737073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0032810 con 0x7f50000d0ee0 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f5006dd5640 1 -- 172.21.15.28:0/1555737073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000d0ee0 msgr2=0x7f50000affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f5006dd5640 1 --2- 172.21.15.28:0/1555737073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000d0ee0 0x7f50000affc0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f50000af410 tx=0x7f4ff002a1c0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f5006dd5640 1 -- 172.21.15.28:0/1555737073 shutdown_connections 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f5006dd5640 1 --2- 172.21.15.28:0/1555737073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000d0ee0 0x7f50000affc0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.654+0000 7f5006dd5640 1 -- 172.21.15.28:0/1555737073 >> 172.21.15.28:0/1555737073 conn(0x7f50000742c0 msgr2=0x7f500006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.655+0000 7f5006dd5640 1 -- 172.21.15.28:0/1555737073 shutdown_connections 2024-04-25T19:12:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.655+0000 7f5006dd5640 1 -- 172.21.15.28:0/1555737073 wait complete. 2024-04-25T19:12:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.656+0000 7f5006dd5640 1 Processor -- start 2024-04-25T19:12:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.656+0000 7f5006dd5640 1 -- start start 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5006dd5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000b5020 0x7f50001570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5006dd5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50000d0ee0 0x7f50001e7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5006dd5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f500015aaa0 0x7f50001e7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5006dd5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f50000b5430 con 0x7f50000b5020 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5006dd5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f50000afaa0 con 0x7f500015aaa0 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5006dd5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f50000b55b0 con 0x7f50000d0ee0 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000b5020 0x7f50001570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000b5020 0x7f50001570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56336/0 (socket says 172.21.15.28:56336) 2024-04-25T19:12:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 -- 172.21.15.28:0/60359254 learned_addr learned my addr 172.21.15.28:0/60359254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5005372640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f500015aaa0 0x7f50001e7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f4ff7fff640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50000d0ee0 0x7f50001e7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 -- 172.21.15.28:0/60359254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50000d0ee0 msgr2=0x7f50001e7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50000d0ee0 0x7f50001e7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 -- 172.21.15.28:0/60359254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f500015aaa0 msgr2=0x7f50001e7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f500015aaa0 0x7f50001e7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 -- 172.21.15.28:0/60359254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5000144720 con 0x7f50000b5020 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.658+0000 7f5004b71640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000b5020 0x7f50001570c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00c430 tx=0x7f4ffc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.659+0000 7f4ff57fa640 1 -- 172.21.15.28:0/60359254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc0074e0 con 0x7f50000b5020 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.659+0000 7f4ff57fa640 1 -- 172.21.15.28:0/60359254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ffc0047e0 con 0x7f50000b5020 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.659+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f500009e030 con 0x7f50000b5020 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.659+0000 7f4ff57fa640 1 -- 172.21.15.28:0/60359254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ffc004bd0 con 0x7f50000b5020 2024-04-25T19:12:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.659+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f500009e5f0 con 0x7f50000b5020 2024-04-25T19:12:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.660+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5000050740 con 0x7f50000b5020 2024-04-25T19:12:17.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.660+0000 7f4ff57fa640 1 -- 172.21.15.28:0/60359254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4ffc011070 con 0x7f50000b5020 2024-04-25T19:12:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.661+0000 7f4ff57fa640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4fec0744e0 0x7f4fec0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.661+0000 7f4ff57fa640 1 -- 172.21.15.28:0/60359254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4ffc095110 con 0x7f50000b5020 2024-04-25T19:12:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.661+0000 7f4ff7fff640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4fec0744e0 0x7f4fec0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.661+0000 7f4ff7fff640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4fec0744e0 0x7f4fec0769f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0034000 tx=0x7f4ff002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.665+0000 7f4ff57fa640 1 -- 172.21.15.28:0/60359254 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ffc05ee50 con 0x7f50000b5020 2024-04-25T19:12:17.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.851+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5000070f20 con 0x7f50000b5020 2024-04-25T19:12:17.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.851+0000 7f4ff57fa640 1 -- 172.21.15.28:0/60359254 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f4ffc063150 con 0x7f50000b5020 2024-04-25T19:12:17.853 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:17.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.853+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4fec0744e0 msgr2=0x7f4fec0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.853+0000 7f5006dd5640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4fec0744e0 0x7f4fec0769f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0034000 tx=0x7f4ff002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.853+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000b5020 msgr2=0x7f50001570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.853+0000 7f5006dd5640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000b5020 0x7f50001570c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4ffc00c430 tx=0x7f4ffc00c8f0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.854+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 shutdown_connections 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.854+0000 7f5006dd5640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4fec0744e0 0x7f4fec0769f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.854+0000 7f5006dd5640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f500015aaa0 0x7f50001e7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.854+0000 7f5006dd5640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f50000d0ee0 0x7f50001e7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.854+0000 7f5006dd5640 1 --2- 172.21.15.28:0/60359254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50000b5020 0x7f50001570c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.854+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 >> 172.21.15.28:0/60359254 conn(0x7f50000742c0 msgr2=0x7f50000cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.854+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 shutdown_connections 2024-04-25T19:12:17.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.854+0000 7f5006dd5640 1 -- 172.21.15.28:0/60359254 wait complete. 2024-04-25T19:12:17.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.981+0000 7fc6f39b2640 1 Processor -- start 2024-04-25T19:12:17.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.981+0000 7fc6f39b2640 1 -- start start 2024-04-25T19:12:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f39b2640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc6ec09caf0 0x7fc6ec0d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f39b2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6ec09cf00 0x7fc6ec0ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f39b2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6ec0d1ce0 con 0x7fc6ec0d2670 2024-04-25T19:12:17.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f39b2640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ec0b8fa0 con 0x7fc6ec09cf00 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f39b2640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fc6ec0cf8d0 con 0x7fc6ec09caf0 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f1f4f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6ec09cf00 0x7fc6ec0ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f1f4f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6ec09cf00 0x7fc6ec0ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56344/0 (socket says 172.21.15.28:56344) 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f1f4f640 1 -- 172.21.15.28:0/1954723958 learned_addr learned my addr 172.21.15.28:0/1954723958 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f174e640 1 --2- 172.21.15.28:0/1954723958 >> v2:172.21.15.53:3300/0 conn(0x7fc6ec09caf0 0x7fc6ec0d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f1f4f640 1 -- 172.21.15.28:0/1954723958 >> v2:172.21.15.53:3300/0 conn(0x7fc6ec09caf0 msgr2=0x7fc6ec0d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f1f4f640 1 --2- 172.21.15.28:0/1954723958 >> v2:172.21.15.53:3300/0 conn(0x7fc6ec09caf0 0x7fc6ec0d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f1f4f640 1 -- 172.21.15.28:0/1954723958 >> v1:172.21.15.53:6789/0 conn(0x7fc6ec0d2670 legacy=0x7fc6ec0b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.983+0000 7fc6f1f4f640 1 -- 172.21.15.28:0/1954723958 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec0cfbb0 con 0x7fc6ec09cf00 2024-04-25T19:12:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.984+0000 7fc6f1f4f640 1 --2- 172.21.15.28:0/1954723958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec09cf00 0x7fc6ec0ce720 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec08cd70 tx=0x7fc6dc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664e23028e821768 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.984+0000 7fc6ebfff640 1 -- 172.21.15.28:0/1954723958 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc036070 con 0x7fc6ec09cf00 2024-04-25T19:12:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.984+0000 7fc6ebfff640 1 -- 172.21.15.28:0/1954723958 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc031600 con 0x7fc6ec09cf00 2024-04-25T19:12:17.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.984+0000 7fc6ebfff640 1 -- 172.21.15.28:0/1954723958 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc0319d0 con 0x7fc6ec09cf00 2024-04-25T19:12:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.985+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1954723958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec09cf00 msgr2=0x7fc6ec0ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.985+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1954723958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec09cf00 0x7fc6ec0ce720 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec08cd70 tx=0x7fc6dc02a6a0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.985+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1954723958 shutdown_connections 2024-04-25T19:12:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.985+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1954723958 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec09cf00 0x7fc6ec0ce720 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.985+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1954723958 >> v2:172.21.15.53:3300/0 conn(0x7fc6ec09caf0 0x7fc6ec0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.985+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1954723958 >> 172.21.15.28:0/1954723958 conn(0x7fc6ec0d03f0 msgr2=0x7fc6ec0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.985+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1954723958 shutdown_connections 2024-04-25T19:12:17.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.985+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1954723958 wait complete. 2024-04-25T19:12:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.987+0000 7fc6f39b2640 1 Processor -- start 2024-04-25T19:12:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.987+0000 7fc6f39b2640 1 -- start start 2024-04-25T19:12:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f39b2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6ec09caf0 0x7fc6ec1dcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f39b2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6ec09cf00 0x7fc6ec0cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f39b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec0d2670 0x7fc6ec0ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f39b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec0b9430 con 0x7fc6ec0d2670 2024-04-25T19:12:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f39b2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc6ec0b78e0 con 0x7fc6ec09caf0 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f39b2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc6ec0d4790 con 0x7fc6ec09cf00 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f1f4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec0d2670 0x7fc6ec0ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f1f4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec0d2670 0x7fc6ec0ca780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56346/0 (socket says 172.21.15.28:56346) 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.988+0000 7fc6f1f4f640 1 -- 172.21.15.28:0/1177464806 learned_addr learned my addr 172.21.15.28:0/1177464806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f174e640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6ec09caf0 0x7fc6ec1dcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f0f4d640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6ec09cf00 0x7fc6ec0cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f1f4f640 1 -- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6ec09cf00 msgr2=0x7fc6ec0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f1f4f640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6ec09cf00 0x7fc6ec0cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f1f4f640 1 -- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6ec09caf0 msgr2=0x7fc6ec1dcbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f1f4f640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6ec09caf0 0x7fc6ec1dcbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f1f4f640 1 -- 172.21.15.28:0/1177464806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6ec1f9ca0 con 0x7fc6ec0d2670 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f0f4d640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6ec09cf00 0x7fc6ec0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f174e640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6ec09caf0 0x7fc6ec1dcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f1f4f640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec0d2670 0x7fc6ec0ca780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc034040 tx=0x7fc6dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6e9ffb640 1 -- 172.21.15.28:0/1177464806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc036050 con 0x7fc6ec0d2670 2024-04-25T19:12:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6e9ffb640 1 -- 172.21.15.28:0/1177464806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6dc042070 con 0x7fc6ec0d2670 2024-04-25T19:12:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6e9ffb640 1 -- 172.21.15.28:0/1177464806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc009510 con 0x7fc6ec0d2670 2024-04-25T19:12:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6ec0cd490 con 0x7fc6ec0d2670 2024-04-25T19:12:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.989+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6ec06d8d0 con 0x7fc6ec0d2670 2024-04-25T19:12:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.990+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6ec050740 con 0x7fc6ec0d2670 2024-04-25T19:12:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.991+0000 7fc6e9ffb640 1 -- 172.21.15.28:0/1177464806 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc6dc007bf0 con 0x7fc6ec0d2670 2024-04-25T19:12:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.991+0000 7fc6e9ffb640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc6d80744b0 0x7fc6d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.991+0000 7fc6e9ffb640 1 -- 172.21.15.28:0/1177464806 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc6dc0b63b0 con 0x7fc6ec0d2670 2024-04-25T19:12:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.991+0000 7fc6f174e640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc6d80744b0 0x7fc6d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.992+0000 7fc6f174e640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc6d80744b0 0x7fc6d80769c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec069f40 tx=0x7fc6e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:17.996+0000 7fc6e9ffb640 1 -- 172.21.15.28:0/1177464806 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6ec1f9ca0 con 0x7fc6ec0d2670 2024-04-25T19:12:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.184+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc6ec0cd1e0 con 0x7fc6ec0d2670 2024-04-25T19:12:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.184+0000 7fc6e9ffb640 1 -- 172.21.15.28:0/1177464806 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fc6dc0807d0 con 0x7fc6ec0d2670 2024-04-25T19:12:18.186 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.186+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc6d80744b0 msgr2=0x7fc6d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.186+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc6d80744b0 0x7fc6d80769c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec069f40 tx=0x7fc6e4008040 comp rx=0 tx=0).stop 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.186+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec0d2670 msgr2=0x7fc6ec0ca780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.186+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec0d2670 0x7fc6ec0ca780 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc034040 tx=0x7fc6dc004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.187+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 shutdown_connections 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.187+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc6d80744b0 0x7fc6d80769c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.187+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6ec0d2670 0x7fc6ec0ca780 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.187+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6ec09cf00 0x7fc6ec0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.187+0000 7fc6f39b2640 1 --2- 172.21.15.28:0/1177464806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6ec09caf0 0x7fc6ec1dcbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.187+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 >> 172.21.15.28:0/1177464806 conn(0x7fc6ec0d03f0 msgr2=0x7fc6ec0714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.187+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 shutdown_connections 2024-04-25T19:12:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.187+0000 7fc6f39b2640 1 -- 172.21.15.28:0/1177464806 wait complete. 2024-04-25T19:12:18.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple' 2024-04-25T19:12:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.309+0000 7f7866c14640 1 Processor -- start 2024-04-25T19:12:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.309+0000 7f7866c14640 1 -- start start 2024-04-25T19:12:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f7866c14640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78600d25c0 0x7f7860160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f7866c14640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78600d29d0 0x7f7860160dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f7866c14640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f786006c710 con 0x7f786006cbf0 2024-04-25T19:12:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f7866c14640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f78600d3790 con 0x7f78600d25c0 2024-04-25T19:12:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f7866c14640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f78600d2ef0 con 0x7f78600d29d0 2024-04-25T19:12:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f78649b0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78600d25c0 0x7f7860160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f78649b0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78600d25c0 0x7f7860160880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56354/0 (socket says 172.21.15.28:56354) 2024-04-25T19:12:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f78649b0640 1 -- 172.21.15.28:0/3775254722 learned_addr learned my addr 172.21.15.28:0/3775254722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f785ffff640 1 --2- 172.21.15.28:0/3775254722 >> v2:172.21.15.53:3300/0 conn(0x7f78600d29d0 0x7f7860160dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f78649b0640 1 -- 172.21.15.28:0/3775254722 >> v2:172.21.15.53:3300/0 conn(0x7f78600d29d0 msgr2=0x7f7860160dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f78649b0640 1 --2- 172.21.15.28:0/3775254722 >> v2:172.21.15.53:3300/0 conn(0x7f78600d29d0 0x7f7860160dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f78649b0640 1 -- 172.21.15.28:0/3775254722 >> v1:172.21.15.53:6789/0 conn(0x7f786006cbf0 legacy=0x7f786006d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f78649b0640 1 -- 172.21.15.28:0/3775254722 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78600d31d0 con 0x7f78600d25c0 2024-04-25T19:12:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.311+0000 7f785ffff640 1 --2- 172.21.15.28:0/3775254722 >> v2:172.21.15.53:3300/0 conn(0x7f78600d29d0 0x7f7860160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.312+0000 7f78649b0640 1 --2- 172.21.15.28:0/3775254722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78600d25c0 0x7f7860160880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7860072ce0 tx=0x7f785802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad89f6c34329dc90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.312+0000 7f785f7fe640 1 -- 172.21.15.28:0/3775254722 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858037070 con 0x7f78600d25c0 2024-04-25T19:12:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.312+0000 7f785f7fe640 1 -- 172.21.15.28:0/3775254722 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7858032440 con 0x7f78600d25c0 2024-04-25T19:12:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.312+0000 7f785f7fe640 1 -- 172.21.15.28:0/3775254722 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858032810 con 0x7f78600d25c0 2024-04-25T19:12:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.313+0000 7f7866c14640 1 -- 172.21.15.28:0/3775254722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78600d25c0 msgr2=0x7f7860160880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:18.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.313+0000 7f7866c14640 1 --2- 172.21.15.28:0/3775254722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78600d25c0 0x7f7860160880 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f7860072ce0 tx=0x7f785802a200 comp rx=0 tx=0).stop 2024-04-25T19:12:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.313+0000 7f7866c14640 1 -- 172.21.15.28:0/3775254722 shutdown_connections 2024-04-25T19:12:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.313+0000 7f7866c14640 1 --2- 172.21.15.28:0/3775254722 >> v2:172.21.15.53:3300/0 conn(0x7f78600d29d0 0x7f7860160dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.313+0000 7f7866c14640 1 --2- 172.21.15.28:0/3775254722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78600d25c0 0x7f7860160880 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.313+0000 7f7866c14640 1 -- 172.21.15.28:0/3775254722 >> 172.21.15.28:0/3775254722 conn(0x7f78600b7f50 msgr2=0x7f78600b76b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.313+0000 7f7866c14640 1 -- 172.21.15.28:0/3775254722 shutdown_connections 2024-04-25T19:12:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.313+0000 7f7866c14640 1 -- 172.21.15.28:0/3775254722 wait complete. 2024-04-25T19:12:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.315+0000 7f7866c14640 1 Processor -- start 2024-04-25T19:12:18.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.315+0000 7f7866c14640 1 -- start start 2024-04-25T19:12:18.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f7866c14640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786006cbf0 0x7f78600ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:18.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f7866c14640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78600d25c0 0x7f78600cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f7866c14640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78600d29d0 0x7f78601f9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f7866c14640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7860161480 con 0x7f786006cbf0 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f7866c14640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7860161300 con 0x7f78600d25c0 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f7866c14640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f786006edc0 con 0x7f78600d29d0 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f78649b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786006cbf0 0x7f78600ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f78649b0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786006cbf0 0x7f78600ccba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56358/0 (socket says 172.21.15.28:56358) 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f78649b0640 1 -- 172.21.15.28:0/1160259699 learned_addr learned my addr 172.21.15.28:0/1160259699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.316+0000 7f785ffff640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78600d25c0 0x7f78600cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f78651b1640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78600d29d0 0x7f78601f9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f78649b0640 1 -- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78600d29d0 msgr2=0x7f78601f9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f78649b0640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78600d29d0 0x7f78601f9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f78649b0640 1 -- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78600d25c0 msgr2=0x7f78600cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f78649b0640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78600d25c0 0x7f78600cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:18.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f78649b0640 1 -- 172.21.15.28:0/1160259699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786014f690 con 0x7f786006cbf0 2024-04-25T19:12:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f785ffff640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78600d25c0 0x7f78600cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f78649b0640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786006cbf0 0x7f78600ccba0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f785802a240 tx=0x7f78580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f785d7fa640 1 -- 172.21.15.28:0/1160259699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858041070 con 0x7f786006cbf0 2024-04-25T19:12:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f785d7fa640 1 -- 172.21.15.28:0/1160259699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785803c870 con 0x7f786006cbf0 2024-04-25T19:12:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786014f970 con 0x7f786006cbf0 2024-04-25T19:12:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f785d7fa640 1 -- 172.21.15.28:0/1160259699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7858037040 con 0x7f786006cbf0 2024-04-25T19:12:18.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.317+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786014ff30 con 0x7f786006cbf0 2024-04-25T19:12:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.318+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786006a100 con 0x7f786006cbf0 2024-04-25T19:12:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.319+0000 7f785d7fa640 1 -- 172.21.15.28:0/1160259699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7858044070 con 0x7f786006cbf0 2024-04-25T19:12:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.319+0000 7f785d7fa640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78400744b0 0x7f78400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.319+0000 7f785ffff640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78400744b0 0x7f78400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.320+0000 7f785d7fa640 1 -- 172.21.15.28:0/1160259699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f78580b62f0 con 0x7f786006cbf0 2024-04-25T19:12:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.320+0000 7f785ffff640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78400744b0 0x7f78400769c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f78600705f0 tx=0x7f78500074d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.324+0000 7f785d7fa640 1 -- 172.21.15.28:0/1160259699 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7858080680 con 0x7f786006cbf0 2024-04-25T19:12:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:18.581+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]} v 0) -- 0x7f78601602a0 con 0x7f786006cbf0 2024-04-25T19:12:19.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.096+0000 7f785d7fa640 1 -- 172.21.15.28:0/1160259699 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]}]=0 v69) ==== 150+0+0 (secure 0 0 0) 0x7f785807ff80 con 0x7f786006cbf0 2024-04-25T19:12:19.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78400744b0 msgr2=0x7f78400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78400744b0 0x7f78400769c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f78600705f0 tx=0x7f78500074d0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786006cbf0 msgr2=0x7f78600ccba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786006cbf0 0x7f78600ccba0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f785802a240 tx=0x7f78580046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 shutdown_connections 2024-04-25T19:12:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78400744b0 0x7f78400769c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78600d29d0 0x7f78601f9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78600d25c0 0x7f78600cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 --2- 172.21.15.28:0/1160259699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786006cbf0 0x7f78600ccba0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 >> 172.21.15.28:0/1160259699 conn(0x7f78600b7f50 msgr2=0x7f78600b4c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 shutdown_connections 2024-04-25T19:12:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.100+0000 7f7866c14640 1 -- 172.21.15.28:0/1160259699 wait complete. 2024-04-25T19:12:19.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T19:12:19.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.235+0000 7f8ee09a8640 1 Processor -- start 2024-04-25T19:12:19.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.235+0000 7f8ee09a8640 1 -- start start 2024-04-25T19:12:19.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.236+0000 7f8ee09a8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8edc160180 0x7f8edc163c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.236+0000 7f8ee09a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d06c0 0x7f8edc0b9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.236+0000 7f8ee09a8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8edc164bb0 con 0x7f8edc0d2550 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.236+0000 7f8ee09a8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8edc0d4510 con 0x7f8edc160180 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.236+0000 7f8ee09a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8edc164310 con 0x7f8edc0d06c0 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.236+0000 7f8eda59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8edc160180 0x7f8edc163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.236+0000 7f8eda59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8edc160180 0x7f8edc163c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56364/0 (socket says 172.21.15.28:56364) 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.236+0000 7f8eda59c640 1 -- 172.21.15.28:0/2250846758 learned_addr learned my addr 172.21.15.28:0/2250846758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.237+0000 7f8ed9d9b640 1 --2- 172.21.15.28:0/2250846758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d06c0 0x7f8edc0b9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.237+0000 7f8eda59c640 1 -- 172.21.15.28:0/2250846758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d06c0 msgr2=0x7f8edc0b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.237+0000 7f8eda59c640 1 --2- 172.21.15.28:0/2250846758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d06c0 0x7f8edc0b9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.237+0000 7f8eda59c640 1 -- 172.21.15.28:0/2250846758 >> v1:172.21.15.53:6789/0 conn(0x7f8edc0d2550 legacy=0x7f8edc0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.237+0000 7f8eda59c640 1 -- 172.21.15.28:0/2250846758 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc1645f0 con 0x7f8edc160180 2024-04-25T19:12:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.237+0000 7f8eda59c640 1 --2- 172.21.15.28:0/2250846758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc163c80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0d4690 tx=0x7f8ecc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fd981828259e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ed959a640 1 -- 172.21.15.28:0/2250846758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc037070 con 0x7f8edc160180 2024-04-25T19:12:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ed959a640 1 -- 172.21.15.28:0/2250846758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ecc032440 con 0x7f8edc160180 2024-04-25T19:12:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ed959a640 1 -- 172.21.15.28:0/2250846758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc032810 con 0x7f8edc160180 2024-04-25T19:12:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ee09a8640 1 -- 172.21.15.28:0/2250846758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 msgr2=0x7f8edc163c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/2250846758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc163c80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8edc0d4690 tx=0x7f8ecc02a250 comp rx=0 tx=0).stop 2024-04-25T19:12:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ee09a8640 1 -- 172.21.15.28:0/2250846758 shutdown_connections 2024-04-25T19:12:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/2250846758 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d06c0 0x7f8edc0b9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/2250846758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc163c80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ee09a8640 1 -- 172.21.15.28:0/2250846758 >> 172.21.15.28:0/2250846758 conn(0x7f8edc06c750 msgr2=0x7f8edc06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ee09a8640 1 -- 172.21.15.28:0/2250846758 shutdown_connections 2024-04-25T19:12:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.238+0000 7f8ee09a8640 1 -- 172.21.15.28:0/2250846758 wait complete. 2024-04-25T19:12:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.240+0000 7f8ee09a8640 1 Processor -- start 2024-04-25T19:12:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.240+0000 7f8ee09a8640 1 -- start start 2024-04-25T19:12:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8ee09a8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8edc0d06c0 0x7f8edc1615e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8ee09a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d2550 0x7f8edc0cf070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8ee09a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc0ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8ee09a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0b95d0 con 0x7f8edc160180 2024-04-25T19:12:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8ee09a8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8edc0d0ad0 con 0x7f8edc0d06c0 2024-04-25T19:12:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8ee09a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8edc0b9750 con 0x7f8edc0d2550 2024-04-25T19:12:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc0ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc0ac530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56378/0 (socket says 172.21.15.28:56378) 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 -- 172.21.15.28:0/1372362479 learned_addr learned my addr 172.21.15.28:0/1372362479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8eda59c640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8edc0d06c0 0x7f8edc1615e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8ed9d9b640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d2550 0x7f8edc0cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 -- 172.21.15.28:0/1372362479 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d2550 msgr2=0x7f8edc0cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d2550 0x7f8edc0cf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 -- 172.21.15.28:0/1372362479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8edc0d06c0 msgr2=0x7f8edc1615e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8edc0d06c0 0x7f8edc1615e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 -- 172.21.15.28:0/1372362479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8edc161f50 con 0x7f8edc160180 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.242+0000 7f8edad9d640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc0ac530 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400c6e0 tx=0x7f8ed400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.243+0000 7f8ecaffd640 1 -- 172.21.15.28:0/1372362479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4003b70 con 0x7f8edc160180 2024-04-25T19:12:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.243+0000 7f8ecaffd640 1 -- 172.21.15.28:0/1372362479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ed4003d30 con 0x7f8edc160180 2024-04-25T19:12:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.243+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8edc162230 con 0x7f8edc160180 2024-04-25T19:12:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.243+0000 7f8ecaffd640 1 -- 172.21.15.28:0/1372362479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4005310 con 0x7f8edc160180 2024-04-25T19:12:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.243+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8edc1627f0 con 0x7f8edc160180 2024-04-25T19:12:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.244+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8edc161cc0 con 0x7f8edc160180 2024-04-25T19:12:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.244+0000 7f8ecaffd640 1 -- 172.21.15.28:0/1372362479 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8ed40054b0 con 0x7f8edc160180 2024-04-25T19:12:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.245+0000 7f8ecaffd640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ebc0744e0 0x7f8ebc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.245+0000 7f8ecaffd640 1 -- 172.21.15.28:0/1372362479 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8ed40954c0 con 0x7f8edc160180 2024-04-25T19:12:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.245+0000 7f8eda59c640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ebc0744e0 0x7f8ebc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.245+0000 7f8eda59c640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ebc0744e0 0x7f8ebc0769f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc034000 tx=0x7f8ecc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.249+0000 7f8ecaffd640 1 -- 172.21.15.28:0/1372362479 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ed405f0e0 con 0x7f8edc160180 2024-04-25T19:12:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.438+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f8edc050740 con 0x7f8edc160180 2024-04-25T19:12:19.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.441+0000 7f8ecaffd640 1 -- 172.21.15.28:0/1372362479 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f8ed40633e0 con 0x7f8edc160180 2024-04-25T19:12:19.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ebc0744e0 msgr2=0x7f8ebc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ebc0744e0 0x7f8ebc0769f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc034000 tx=0x7f8ecc02e000 comp rx=0 tx=0).stop 2024-04-25T19:12:19.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 msgr2=0x7f8edc0ac530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc0ac530 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400c6e0 tx=0x7f8ed400cba0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 shutdown_connections 2024-04-25T19:12:19.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ebc0744e0 0x7f8ebc0769f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8edc160180 0x7f8edc0ac530 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8edc0d2550 0x7f8edc0cf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 --2- 172.21.15.28:0/1372362479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8edc0d06c0 0x7f8edc1615e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 >> 172.21.15.28:0/1372362479 conn(0x7f8edc06c750 msgr2=0x7f8edc0b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 shutdown_connections 2024-04-25T19:12:19.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.443+0000 7f8ee09a8640 1 -- 172.21.15.28:0/1372362479 wait complete. 2024-04-25T19:12:19.456 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T19:12:19.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T19:12:19.497 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T19:12:19.540 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T19:12:19.541 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:19.541 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:19.556 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:19.556 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:19.612 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:19.612 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:19.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:19.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.818+0000 7f350575f640 1 Processor -- start 2024-04-25T19:12:19.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.818+0000 7f350575f640 1 -- start start 2024-04-25T19:12:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.819+0000 7f350575f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35000b5130 0x7f3500161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.819+0000 7f350575f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f350015f7d0 0x7f350015fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f350575f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3500163e80 con 0x7f350006ded0 2024-04-25T19:12:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f350575f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f3500163c30 con 0x7f35000b5130 2024-04-25T19:12:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f350575f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f3500164a90 con 0x7f350015f7d0 2024-04-25T19:12:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f34feffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35000b5130 0x7f3500161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f34feffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35000b5130 0x7f3500161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56390/0 (socket says 172.21.15.28:56390) 2024-04-25T19:12:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f34feffd640 1 -- 172.21.15.28:0/1857308782 learned_addr learned my addr 172.21.15.28:0/1857308782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:19.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f34ff7fe640 1 --2- 172.21.15.28:0/1857308782 >> v2:172.21.15.53:3300/0 conn(0x7f350015f7d0 0x7f350015fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f34feffd640 1 -- 172.21.15.28:0/1857308782 >> v2:172.21.15.53:3300/0 conn(0x7f350015f7d0 msgr2=0x7f350015fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:19.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f34feffd640 1 --2- 172.21.15.28:0/1857308782 >> v2:172.21.15.53:3300/0 conn(0x7f350015f7d0 0x7f350015fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f34feffd640 1 -- 172.21.15.28:0/1857308782 >> v1:172.21.15.53:6789/0 conn(0x7f350006ded0 legacy=0x7f350006e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:19.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.820+0000 7f34feffd640 1 -- 172.21.15.28:0/1857308782 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3500164160 con 0x7f35000b5130 2024-04-25T19:12:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.821+0000 7f34feffd640 1 --2- 172.21.15.28:0/1857308782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500161130 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3500163b70 tx=0x7f34f002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a0cfcbb2d4e4b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.821+0000 7f34fdffb640 1 -- 172.21.15.28:0/1857308782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0037070 con 0x7f35000b5130 2024-04-25T19:12:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.821+0000 7f34fdffb640 1 -- 172.21.15.28:0/1857308782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f0032440 con 0x7f35000b5130 2024-04-25T19:12:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.821+0000 7f34fdffb640 1 -- 172.21.15.28:0/1857308782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0032810 con 0x7f35000b5130 2024-04-25T19:12:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.822+0000 7f350575f640 1 -- 172.21.15.28:0/1857308782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 msgr2=0x7f3500161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.822+0000 7f350575f640 1 --2- 172.21.15.28:0/1857308782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500161130 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3500163b70 tx=0x7f34f002a230 comp rx=0 tx=0).stop 2024-04-25T19:12:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.822+0000 7f350575f640 1 -- 172.21.15.28:0/1857308782 shutdown_connections 2024-04-25T19:12:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.822+0000 7f350575f640 1 --2- 172.21.15.28:0/1857308782 >> v2:172.21.15.53:3300/0 conn(0x7f350015f7d0 0x7f350015fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.822+0000 7f350575f640 1 --2- 172.21.15.28:0/1857308782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500161130 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.822+0000 7f350575f640 1 -- 172.21.15.28:0/1857308782 >> 172.21.15.28:0/1857308782 conn(0x7f35000b7ad0 msgr2=0x7f35000b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:19.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.822+0000 7f350575f640 1 -- 172.21.15.28:0/1857308782 shutdown_connections 2024-04-25T19:12:19.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.822+0000 7f350575f640 1 -- 172.21.15.28:0/1857308782 wait complete. 2024-04-25T19:12:19.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.824+0000 7f350575f640 1 Processor -- start 2024-04-25T19:12:19.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.824+0000 7f350575f640 1 -- start start 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f350575f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f350006ded0 0x7f350015cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f350575f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f350575f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f350015f7d0 0x7f35001f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f350575f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3500160120 con 0x7f35000b5130 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f350575f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3500164720 con 0x7f350006ded0 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f350575f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f35001602a0 con 0x7f350015f7d0 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34fe7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34fe7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56400/0 (socket says 172.21.15.28:56400) 2024-04-25T19:12:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34fe7fc640 1 -- 172.21.15.28:0/798775371 learned_addr learned my addr 172.21.15.28:0/798775371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34ff7fe640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f350015f7d0 0x7f35001f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34feffd640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f350006ded0 0x7f350015cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34fe7fc640 1 -- 172.21.15.28:0/798775371 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f350015f7d0 msgr2=0x7f35001f5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34fe7fc640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f350015f7d0 0x7f35001f5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34fe7fc640 1 -- 172.21.15.28:0/798775371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f350006ded0 msgr2=0x7f350015cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34fe7fc640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f350006ded0 0x7f350015cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.825+0000 7f34fe7fc640 1 -- 172.21.15.28:0/798775371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350015c270 con 0x7f35000b5130 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.826+0000 7f34fe7fc640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500157d30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f34f400b3f0 tx=0x7f34f400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.826+0000 7f34e77fe640 1 -- 172.21.15.28:0/798775371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4015020 con 0x7f35000b5130 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.826+0000 7f34e77fe640 1 -- 172.21.15.28:0/798775371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f4011830 con 0x7f35000b5130 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.826+0000 7f34e77fe640 1 -- 172.21.15.28:0/798775371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f4011c20 con 0x7f35000b5130 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.826+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3500158900 con 0x7f35000b5130 2024-04-25T19:12:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.826+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f350015c7f0 con 0x7f35000b5130 2024-04-25T19:12:19.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.827+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3500050740 con 0x7f35000b5130 2024-04-25T19:12:19.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.827+0000 7f34e77fe640 1 -- 172.21.15.28:0/798775371 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f34f4018020 con 0x7f35000b5130 2024-04-25T19:12:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.828+0000 7f34e77fe640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f34ec0744b0 0x7f34ec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.828+0000 7f34e77fe640 1 -- 172.21.15.28:0/798775371 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f34f40951f0 con 0x7f35000b5130 2024-04-25T19:12:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.828+0000 7f34feffd640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f34ec0744b0 0x7f34ec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.829+0000 7f34feffd640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f34ec0744b0 0x7f34ec0769c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f34f0034000 tx=0x7f34f002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:19.833+0000 7f34e77fe640 1 -- 172.21.15.28:0/798775371 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34f401f030 con 0x7f35000b5130 2024-04-25T19:12:20.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.019+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f35000d2fe0 con 0x7f35000b5130 2024-04-25T19:12:20.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.021+0000 7f34e77fe640 1 -- 172.21.15.28:0/798775371 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f34f405f610 con 0x7f35000b5130 2024-04-25T19:12:20.023 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:20.023 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:20.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.022+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f34ec0744b0 msgr2=0x7f34ec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.022+0000 7f350575f640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f34ec0744b0 0x7f34ec0769c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f34f0034000 tx=0x7f34f002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.022+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 msgr2=0x7f3500157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.022+0000 7f350575f640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500157d30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f34f400b3f0 tx=0x7f34f400b8b0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.023+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 shutdown_connections 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.023+0000 7f350575f640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f34ec0744b0 0x7f34ec0769c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.023+0000 7f350575f640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f350015f7d0 0x7f35001f5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.023+0000 7f350575f640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35000b5130 0x7f3500157d30 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.023+0000 7f350575f640 1 --2- 172.21.15.28:0/798775371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f350006ded0 0x7f350015cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.023+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 >> 172.21.15.28:0/798775371 conn(0x7f35000b7ad0 msgr2=0x7f35000b9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.023+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 shutdown_connections 2024-04-25T19:12:20.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.023+0000 7f350575f640 1 -- 172.21.15.28:0/798775371 wait complete. 2024-04-25T19:12:20.036 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:20.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.150+0000 7f301dad1640 1 Processor -- start 2024-04-25T19:12:20.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.150+0000 7f301dad1640 1 -- start start 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f301dad1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 0x7f3018167520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f301dad1640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f301815db10 con 0x7f301806fc30 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f301dad1640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30180af3c0 con 0x7f3018070b60 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f301dad1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f301815e0d0 con 0x7f301815f570 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f3016ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f301806fc30 0x7f3018070040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50624/0 (socket says 172.21.15.28:50624) 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f3016ffd640 1 -- 172.21.15.28:0/56283271 learned_addr learned my addr 172.21.15.28:0/56283271 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f3017fff640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 0x7f3018167520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1247488728 0 0) 0x7f301815db10 con 0x7f301806fc30 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.152+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3000003520 con 0x7f301806fc30 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3086454768 0 0) 0x7f3000003520 con 0x7f301806fc30 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 msgr2=0x7f3018167520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 0x7f3018167520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 >> v1:172.21.15.53:6789/0 conn(0x7f3018070b60 legacy=0x7f301815d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30180af6a0 con 0x7f301806fc30 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f300c001690 con 0x7f301806fc30 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 >> v1:172.21.15.28:6789/0 conn(0x7f301806fc30 legacy=0x7f3018070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f3000003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3000004800 0x7f3000004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 0x7f30000052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f301815e0d0 con 0x7f30000039e0 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f300000bb80 con 0x7f3000004800 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f300000bd80 con 0x7f301815f570 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30177fe640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f3000003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f3016ffd640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3000004800 0x7f3000004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f3017fff640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 0x7f30000052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30177fe640 1 -- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 msgr2=0x7f30000052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30177fe640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 0x7f30000052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30177fe640 1 -- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3000004800 msgr2=0x7f3000004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30177fe640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3000004800 0x7f3000004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f30177fe640 1 -- 172.21.15.28:0/56283271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3000003520 con 0x7f30000039e0 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f3017fff640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f301815f570 0x7f30000052d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.153+0000 7f3016ffd640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3000004800 0x7f3000004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.154+0000 7f30177fe640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f3000003e50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3008001750 tx=0x7f3000005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806c1dc8f28084ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.154+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008001370 con 0x7f30000039e0 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.154+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30080052d0 con 0x7f30000039e0 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.154+0000 7f30167fc640 1 -- 172.21.15.28:0/56283271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300802abe0 con 0x7f30000039e0 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.155+0000 7f301dad1640 1 -- 172.21.15.28:0/56283271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 msgr2=0x7f3000003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.155+0000 7f301dad1640 1 --2- 172.21.15.28:0/56283271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f3000003e50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3008001750 tx=0x7f3000005830 comp rx=0 tx=0).stop 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.155+0000 7f3017fff640 1 -- 172.21.15.28:0/56283271 reap_dead start 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.155+0000 7f301dad1640 1 -- 172.21.15.28:0/56283271 shutdown_connections 2024-04-25T19:12:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.155+0000 7f301dad1640 1 -- 172.21.15.28:0/56283271 >> 172.21.15.28:0/56283271 conn(0x7f30180742c0 msgr2=0x7f301806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.155+0000 7f301dad1640 1 -- 172.21.15.28:0/56283271 shutdown_connections 2024-04-25T19:12:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.155+0000 7f301dad1640 1 -- 172.21.15.28:0/56283271 wait complete. 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.156+0000 7f301dad1640 1 Processor -- start 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.157+0000 7f301dad1640 1 -- start start 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f301dad1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f301809e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f301dad1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30181556e0 0x7f30181488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f301dad1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3018148e00 0x7f3018149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f301dad1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f301815dd60 con 0x7f30000039e0 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f301dad1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3018153b00 con 0x7f30181556e0 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f301dad1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f301806f3a0 con 0x7f3018148e00 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f301809e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f301809e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56430/0 (socket says 172.21.15.28:56430) 2024-04-25T19:12:20.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 -- 172.21.15.28:0/1158934347 learned_addr learned my addr 172.21.15.28:0/1158934347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f3017fff640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3018148e00 0x7f3018149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f3016ffd640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30181556e0 0x7f30181488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 -- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3018148e00 msgr2=0x7f3018149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3018148e00 0x7f3018149210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 -- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30181556e0 msgr2=0x7f30181488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30181556e0 0x7f30181488c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 -- 172.21.15.28:0/1158934347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3018155e50 con 0x7f30000039e0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f3017fff640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3018148e00 0x7f3018149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.158+0000 7f30177fe640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f301809e610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3008001870 tx=0x7f3008001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.159+0000 7f2ffffff640 1 -- 172.21.15.28:0/1158934347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008037040 con 0x7f30000039e0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.159+0000 7f2ffffff640 1 -- 172.21.15.28:0/1158934347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3008005480 con 0x7f30000039e0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.159+0000 7f2ffffff640 1 -- 172.21.15.28:0/1158934347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008007a30 con 0x7f30000039e0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.159+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3018156130 con 0x7f30000039e0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.159+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30181566f0 con 0x7f30000039e0 2024-04-25T19:12:20.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.160+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3018050740 con 0x7f30000039e0 2024-04-25T19:12:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.161+0000 7f2ffffff640 1 -- 172.21.15.28:0/1158934347 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3008044020 con 0x7f30000039e0 2024-04-25T19:12:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.161+0000 7f2ffffff640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f30040744b0 0x7f30040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.161+0000 7f2ffffff640 1 -- 172.21.15.28:0/1158934347 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f30080b5440 con 0x7f30000039e0 2024-04-25T19:12:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.161+0000 7f3016ffd640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f30040744b0 0x7f30040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.162+0000 7f3016ffd640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f30040744b0 0x7f30040769c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f300c001220 tx=0x7f300c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.166+0000 7f2ffffff640 1 -- 172.21.15.28:0/1158934347 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f300807f7d0 con 0x7f30000039e0 2024-04-25T19:12:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.349+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3018070940 con 0x7f30000039e0 2024-04-25T19:12:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.349+0000 7f2ffffff640 1 -- 172.21.15.28:0/1158934347 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f300807f0d0 con 0x7f30000039e0 2024-04-25T19:12:20.351 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f30040744b0 msgr2=0x7f30040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f30040744b0 0x7f30040769c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f300c001220 tx=0x7f300c009040 comp rx=0 tx=0).stop 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 msgr2=0x7f301809e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f301809e610 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3008001870 tx=0x7f3008001120 comp rx=0 tx=0).stop 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 shutdown_connections 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f30040744b0 0x7f30040769c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3018148e00 0x7f3018149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30181556e0 0x7f30181488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 --2- 172.21.15.28:0/1158934347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000039e0 0x7f301809e610 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.351+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 >> 172.21.15.28:0/1158934347 conn(0x7f30180742c0 msgr2=0x7f301806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.352+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 shutdown_connections 2024-04-25T19:12:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.352+0000 7f301dad1640 1 -- 172.21.15.28:0/1158934347 wait complete. 2024-04-25T19:12:20.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.477+0000 7ff468014640 1 Processor -- start 2024-04-25T19:12:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.477+0000 7ff468014640 1 -- start start 2024-04-25T19:12:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff468014640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff4600d2d10 0x7ff4600d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff468014640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff4600b3d00 0x7ff4600b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff468014640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4600b92a0 con 0x7ff4600b4650 2024-04-25T19:12:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff468014640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7ff46009cea0 con 0x7ff4600d2d10 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff468014640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7ff46006e8e0 con 0x7ff4600b3d00 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff465db0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff4600d2d10 0x7ff4600d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff4665b1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4600b4650 0x7ff46006d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50626/0 (socket says 172.21.15.28:50626) 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff4665b1640 1 -- 172.21.15.28:0/3834822285 learned_addr learned my addr 172.21.15.28:0/3834822285 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff4655af640 1 --2- 172.21.15.28:0/3834822285 >> v2:172.21.15.53:3300/0 conn(0x7ff4600b3d00 0x7ff4600b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff465db0640 1 -- 172.21.15.28:0/3834822285 >> v2:172.21.15.53:3300/0 conn(0x7ff4600b3d00 msgr2=0x7ff4600b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff465db0640 1 --2- 172.21.15.28:0/3834822285 >> v2:172.21.15.53:3300/0 conn(0x7ff4600b3d00 0x7ff4600b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff465db0640 1 -- 172.21.15.28:0/3834822285 >> v1:172.21.15.28:6789/0 conn(0x7ff4600b4650 legacy=0x7ff46006d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff465db0640 1 -- 172.21.15.28:0/3834822285 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4600b8ee0 con 0x7ff4600d2d10 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.479+0000 7ff464dae640 1 -- 172.21.15.28:0/3834822285 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2079039859 0 0) 0x7ff4600b92a0 con 0x7ff4600b4650 2024-04-25T19:12:20.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.480+0000 7ff465db0640 1 --2- 172.21.15.28:0/3834822285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600d2d10 0x7ff4600d21c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff46006f7c0 tx=0x7ff45802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23e5c916f6538dc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.480+0000 7ff464dae640 1 -- 172.21.15.28:0/3834822285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458037070 con 0x7ff4600d2d10 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.480+0000 7ff464dae640 1 -- 172.21.15.28:0/3834822285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff458032440 con 0x7ff4600d2d10 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.480+0000 7ff464dae640 1 -- 172.21.15.28:0/3834822285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45802ae60 con 0x7ff4600d2d10 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.481+0000 7ff468014640 1 -- 172.21.15.28:0/3834822285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600d2d10 msgr2=0x7ff4600d21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.481+0000 7ff468014640 1 --2- 172.21.15.28:0/3834822285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600d2d10 0x7ff4600d21c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff46006f7c0 tx=0x7ff45802a110 comp rx=0 tx=0).stop 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.481+0000 7ff468014640 1 -- 172.21.15.28:0/3834822285 shutdown_connections 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.481+0000 7ff468014640 1 --2- 172.21.15.28:0/3834822285 >> v2:172.21.15.53:3300/0 conn(0x7ff4600b3d00 0x7ff4600b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.481+0000 7ff468014640 1 --2- 172.21.15.28:0/3834822285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600d2d10 0x7ff4600d21c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.481+0000 7ff468014640 1 -- 172.21.15.28:0/3834822285 >> 172.21.15.28:0/3834822285 conn(0x7ff4600d0450 msgr2=0x7ff4600d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.481+0000 7ff468014640 1 -- 172.21.15.28:0/3834822285 shutdown_connections 2024-04-25T19:12:20.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.481+0000 7ff468014640 1 -- 172.21.15.28:0/3834822285 wait complete. 2024-04-25T19:12:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.483+0000 7ff468014640 1 Processor -- start 2024-04-25T19:12:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.483+0000 7ff468014640 1 -- start start 2024-04-25T19:12:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.484+0000 7ff468014640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4600b3d00 0x7ff4601dba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.484+0000 7ff468014640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600b4650 0x7ff4600cd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.484+0000 7ff468014640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4600d2d10 0x7ff4601f9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.484+0000 7ff468014640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4600d2700 con 0x7ff4600b4650 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.484+0000 7ff468014640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff4600d3120 con 0x7ff4600d2d10 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.484+0000 7ff468014640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff4600d2880 con 0x7ff4600b3d00 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.484+0000 7ff4655af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600b4650 0x7ff4600cd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4655af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600b4650 0x7ff4600cd290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56450/0 (socket says 172.21.15.28:56450) 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4655af640 1 -- 172.21.15.28:0/1898458668 learned_addr learned my addr 172.21.15.28:0/1898458668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff465db0640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4600b3d00 0x7ff4601dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4665b1640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4600d2d10 0x7ff4601f9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4655af640 1 -- 172.21.15.28:0/1898458668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4600b3d00 msgr2=0x7ff4601dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4655af640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4600b3d00 0x7ff4601dba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4655af640 1 -- 172.21.15.28:0/1898458668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4600d2d10 msgr2=0x7ff4601f9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4655af640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4600d2d10 0x7ff4601f9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4655af640 1 -- 172.21.15.28:0/1898458668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4600c74e0 con 0x7ff4600b4650 2024-04-25T19:12:20.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff4655af640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600b4650 0x7ff4600cd290 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff45400d600 tx=0x7ff45400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff45e7fc640 1 -- 172.21.15.28:0/1898458668 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4540125f0 con 0x7ff4600b4650 2024-04-25T19:12:20.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff45e7fc640 1 -- 172.21.15.28:0/1898458668 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff454019070 con 0x7ff4600b4650 2024-04-25T19:12:20.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff45e7fc640 1 -- 172.21.15.28:0/1898458668 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4540129b0 con 0x7ff4600b4650 2024-04-25T19:12:20.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4600c79e0 con 0x7ff4600b4650 2024-04-25T19:12:20.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.485+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4600c7fa0 con 0x7ff4600b4650 2024-04-25T19:12:20.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.486+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4600c7750 con 0x7ff4600b4650 2024-04-25T19:12:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.487+0000 7ff45e7fc640 1 -- 172.21.15.28:0/1898458668 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff454012b50 con 0x7ff4600b4650 2024-04-25T19:12:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.488+0000 7ff45e7fc640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff44c0744e0 0x7ff44c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.488+0000 7ff465db0640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff44c0744e0 0x7ff44c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.488+0000 7ff45e7fc640 1 -- 172.21.15.28:0/1898458668 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff454095830 con 0x7ff4600b4650 2024-04-25T19:12:20.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.488+0000 7ff465db0640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff44c0744e0 0x7ff44c0769f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff458034000 tx=0x7ff45802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.492+0000 7ff45e7fc640 1 -- 172.21.15.28:0/1898458668 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff45405f560 con 0x7ff4600b4650 2024-04-25T19:12:20.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.681+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff460050740 con 0x7ff4600b4650 2024-04-25T19:12:20.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.682+0000 7ff45e7fc640 1 -- 172.21.15.28:0/1898458668 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7ff454063860 con 0x7ff4600b4650 2024-04-25T19:12:20.684 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff44c0744e0 msgr2=0x7ff44c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff44c0744e0 0x7ff44c0769f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff458034000 tx=0x7ff45802e000 comp rx=0 tx=0).stop 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600b4650 msgr2=0x7ff4600cd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600b4650 0x7ff4600cd290 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff45400d600 tx=0x7ff45400dac0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 shutdown_connections 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff44c0744e0 0x7ff44c0769f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff4600d2d10 0x7ff4601f9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4600b4650 0x7ff4600cd290 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 --2- 172.21.15.28:0/1898458668 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff4600b3d00 0x7ff4601dba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 >> 172.21.15.28:0/1898458668 conn(0x7ff4600d0450 msgr2=0x7ff4600ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:20.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 shutdown_connections 2024-04-25T19:12:20.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.684+0000 7ff468014640 1 -- 172.21.15.28:0/1898458668 wait complete. 2024-04-25T19:12:20.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.810+0000 7fe33f906640 1 Processor -- start 2024-04-25T19:12:20.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.810+0000 7fe33f906640 1 -- start start 2024-04-25T19:12:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.811+0000 7fe33f906640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3380ba930 0x7fe33816ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.811+0000 7fe33f906640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe338160720 con 0x7fe338160310 2024-04-25T19:12:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.811+0000 7fe33f906640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3380d0f40 con 0x7fe3380b79f0 2024-04-25T19:12:20.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.811+0000 7fe33f906640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fe3380d1220 con 0x7fe3380ba930 2024-04-25T19:12:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.811+0000 7fe33dea3640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3380ba930 0x7fe33816ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.811+0000 7fe33d6a2640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe338160310 0x7fe3380d5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36832/0 (socket says 172.21.15.28:36832) 2024-04-25T19:12:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.811+0000 7fe33d6a2640 1 -- 172.21.15.28:0/641633814 learned_addr learned my addr 172.21.15.28:0/641633814 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.812+0000 7fe33dea3640 1 -- 172.21.15.28:0/641633814 >> v1:172.21.15.53:6789/0 conn(0x7fe3380b79f0 legacy=0x7fe3380b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.812+0000 7fe33dea3640 1 -- 172.21.15.28:0/641633814 >> v1:172.21.15.28:6789/0 conn(0x7fe338160310 legacy=0x7fe3380d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.812+0000 7fe33dea3640 1 -- 172.21.15.28:0/641633814 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3380d1500 con 0x7fe3380ba930 2024-04-25T19:12:20.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.812+0000 7fe32ffff640 1 -- 172.21.15.28:0/641633814 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2158963595 0 0) 0x7fe338160720 con 0x7fe338160310 2024-04-25T19:12:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.812+0000 7fe33dea3640 1 --2- 172.21.15.28:0/641633814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3380ba930 0x7fe33816ae70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe330009d90 tx=0x7fe33002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc2cf87f6845c81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.813+0000 7fe32ffff640 1 -- 172.21.15.28:0/641633814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330007740 con 0x7fe3380ba930 2024-04-25T19:12:20.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.813+0000 7fe32ffff640 1 -- 172.21.15.28:0/641633814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3300078e0 con 0x7fe3380ba930 2024-04-25T19:12:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.813+0000 7fe32ffff640 1 -- 172.21.15.28:0/641633814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330007cb0 con 0x7fe3380ba930 2024-04-25T19:12:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.813+0000 7fe33f906640 1 -- 172.21.15.28:0/641633814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3380ba930 msgr2=0x7fe33816ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.813+0000 7fe33f906640 1 --2- 172.21.15.28:0/641633814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3380ba930 0x7fe33816ae70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe330009d90 tx=0x7fe33002a6f0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.813+0000 7fe33f906640 1 -- 172.21.15.28:0/641633814 shutdown_connections 2024-04-25T19:12:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.813+0000 7fe33f906640 1 --2- 172.21.15.28:0/641633814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3380ba930 0x7fe33816ae70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.813+0000 7fe33f906640 1 -- 172.21.15.28:0/641633814 >> 172.21.15.28:0/641633814 conn(0x7fe3380d2a50 msgr2=0x7fe3380d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.814+0000 7fe33f906640 1 -- 172.21.15.28:0/641633814 shutdown_connections 2024-04-25T19:12:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.814+0000 7fe33f906640 1 -- 172.21.15.28:0/641633814 wait complete. 2024-04-25T19:12:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.815+0000 7fe33f906640 1 Processor -- start 2024-04-25T19:12:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.815+0000 7fe33f906640 1 -- start start 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33f906640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3380b79f0 0x7fe33815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33f906640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3380ba930 0x7fe3381e5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33f906640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe338160310 0x7fe3380ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33f906640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe3380cc950 con 0x7fe338160310 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33f906640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe3380d0bf0 con 0x7fe3380ba930 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33f906640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe3380d17e0 con 0x7fe3380b79f0 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe338160310 0x7fe3380ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe338160310 0x7fe3380ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38090/0 (socket says 172.21.15.28:38090) 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 -- 172.21.15.28:0/2630809393 learned_addr learned my addr 172.21.15.28:0/2630809393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33cea1640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3380ba930 0x7fe3381e5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33d6a2640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3380b79f0 0x7fe33815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 -- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3380b79f0 msgr2=0x7fe33815bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3380b79f0 0x7fe33815bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 -- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3380ba930 msgr2=0x7fe3381e5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3380ba930 0x7fe3381e5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 -- 172.21.15.28:0/2630809393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3381e5e10 con 0x7fe338160310 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33d6a2640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3380b79f0 0x7fe33815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe33dea3640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe338160310 0x7fe3380ceca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe330009d90 tx=0x7fe330004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.817+0000 7fe32dffb640 1 -- 172.21.15.28:0/2630809393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330040070 con 0x7fe338160310 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.818+0000 7fe32dffb640 1 -- 172.21.15.28:0/2630809393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe330031c40 con 0x7fe338160310 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.818+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3381e60f0 con 0x7fe338160310 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.818+0000 7fe32dffb640 1 -- 172.21.15.28:0/2630809393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330003990 con 0x7fe338160310 2024-04-25T19:12:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.818+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3381e71e0 con 0x7fe338160310 2024-04-25T19:12:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.819+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3381e5b80 con 0x7fe338160310 2024-04-25T19:12:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.819+0000 7fe32dffb640 1 -- 172.21.15.28:0/2630809393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe330039040 con 0x7fe338160310 2024-04-25T19:12:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.820+0000 7fe32dffb640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe31c0744b0 0x7fe31c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.820+0000 7fe33d6a2640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe31c0744b0 0x7fe31c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.820+0000 7fe32dffb640 1 -- 172.21.15.28:0/2630809393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe3300b69b0 con 0x7fe338160310 2024-04-25T19:12:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.820+0000 7fe33d6a2640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe31c0744b0 0x7fe31c0769c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe334000f60 tx=0x7fe334009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:20.825+0000 7fe32dffb640 1 -- 172.21.15.28:0/2630809393 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe330080d30 con 0x7fe338160310 2024-04-25T19:12:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.009+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe3381e7440 con 0x7fe338160310 2024-04-25T19:12:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.010+0000 7fe32dffb640 1 -- 172.21.15.28:0/2630809393 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fe330080630 con 0x7fe338160310 2024-04-25T19:12:21.012 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe31c0744b0 msgr2=0x7fe31c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe31c0744b0 0x7fe31c0769c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe334000f60 tx=0x7fe334009040 comp rx=0 tx=0).stop 2024-04-25T19:12:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe338160310 msgr2=0x7fe3380ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe338160310 0x7fe3380ceca0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fe330009d90 tx=0x7fe330004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 shutdown_connections 2024-04-25T19:12:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe31c0744b0 0x7fe31c0769c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe338160310 0x7fe3380ceca0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3380ba930 0x7fe3381e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 --2- 172.21.15.28:0/2630809393 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe3380b79f0 0x7fe33815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 >> 172.21.15.28:0/2630809393 conn(0x7fe3380d2a50 msgr2=0x7fe338068600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 shutdown_connections 2024-04-25T19:12:21.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.012+0000 7fe33f906640 1 -- 172.21.15.28:0/2630809393 wait complete. 2024-04-25T19:12:21.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.138+0000 7ff726a84640 1 Processor -- start 2024-04-25T19:12:21.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.138+0000 7ff726a84640 1 -- start start 2024-04-25T19:12:21.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff726a84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009caf0 0x7ff7200d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff726a84640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff72009cf00 0x7ff7200ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff726a84640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7200d1ce0 con 0x7ff7200d2670 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff726a84640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7ff7200b8fa0 con 0x7ff72009cf00 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff726a84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff7200cf8d0 con 0x7ff72009caf0 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff725021640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff72009cf00 0x7ff7200ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff725021640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff72009cf00 0x7ff7200ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38102/0 (socket says 172.21.15.28:38102) 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff71ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff7200d2670 0x7ff7200b71d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36836/0 (socket says 172.21.15.28:36836) 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.139+0000 7ff725021640 1 -- 172.21.15.28:0/3466672863 learned_addr learned my addr 172.21.15.28:0/3466672863 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:21.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.140+0000 7ff724820640 1 --2- 172.21.15.28:0/3466672863 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009caf0 0x7ff7200d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.140+0000 7ff725021640 1 -- 172.21.15.28:0/3466672863 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009caf0 msgr2=0x7ff7200d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.140+0000 7ff725021640 1 --2- 172.21.15.28:0/3466672863 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009caf0 0x7ff7200d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.140+0000 7ff725021640 1 -- 172.21.15.28:0/3466672863 >> v1:172.21.15.28:6789/0 conn(0x7ff7200d2670 legacy=0x7ff7200b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.140+0000 7ff725021640 1 -- 172.21.15.28:0/3466672863 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7200cfbb0 con 0x7ff72009cf00 2024-04-25T19:12:21.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.140+0000 7ff724820640 1 --2- 172.21.15.28:0/3466672863 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009caf0 0x7ff7200d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.140+0000 7ff725021640 1 --2- 172.21.15.28:0/3466672863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009cf00 0x7ff7200ce720 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff718009e50 tx=0x7ff71802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f70f4e3a3f83a48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff71f7fe640 1 -- 172.21.15.28:0/3466672863 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718036070 con 0x7ff72009cf00 2024-04-25T19:12:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff71f7fe640 1 -- 172.21.15.28:0/3466672863 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff718031600 con 0x7ff72009cf00 2024-04-25T19:12:21.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff71f7fe640 1 -- 172.21.15.28:0/3466672863 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7180319d0 con 0x7ff72009cf00 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff726a84640 1 -- 172.21.15.28:0/3466672863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009cf00 msgr2=0x7ff7200ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff726a84640 1 --2- 172.21.15.28:0/3466672863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009cf00 0x7ff7200ce720 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff718009e50 tx=0x7ff71802a680 comp rx=0 tx=0).stop 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff726a84640 1 -- 172.21.15.28:0/3466672863 shutdown_connections 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff726a84640 1 --2- 172.21.15.28:0/3466672863 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009cf00 0x7ff7200ce720 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff726a84640 1 --2- 172.21.15.28:0/3466672863 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009caf0 0x7ff7200d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff726a84640 1 -- 172.21.15.28:0/3466672863 >> 172.21.15.28:0/3466672863 conn(0x7ff7200d03f0 msgr2=0x7ff7200d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff726a84640 1 -- 172.21.15.28:0/3466672863 shutdown_connections 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.141+0000 7ff726a84640 1 -- 172.21.15.28:0/3466672863 wait complete. 2024-04-25T19:12:21.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.143+0000 7ff726a84640 1 Processor -- start 2024-04-25T19:12:21.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.143+0000 7ff726a84640 1 -- start start 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff726a84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009caf0 0x7ff72006eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff726a84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009cf00 0x7ff7201d3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff726a84640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7200d2670 0x7ff7200c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff726a84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7200b9430 con 0x7ff72009caf0 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff726a84640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff7200b78e0 con 0x7ff7200d2670 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff726a84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff7200d4790 con 0x7ff72009cf00 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009caf0 0x7ff72006eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009caf0 0x7ff72006eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38118/0 (socket says 172.21.15.28:38118) 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 -- 172.21.15.28:0/2749283167 learned_addr learned my addr 172.21.15.28:0/2749283167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff71ffff640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009cf00 0x7ff7201d3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff725021640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7200d2670 0x7ff7200c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 -- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009cf00 msgr2=0x7ff7201d3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009cf00 0x7ff7201d3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 -- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7200d2670 msgr2=0x7ff7200c8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7200d2670 0x7ff7200c8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 -- 172.21.15.28:0/2749283167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff72006f000 con 0x7ff72009caf0 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff71ffff640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009cf00 0x7ff7201d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff724820640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009caf0 0x7ff72006eac0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff71400c7c0 tx=0x7ff71400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff71d7fa640 1 -- 172.21.15.28:0/2749283167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714015070 con 0x7ff72009caf0 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff71d7fa640 1 -- 172.21.15.28:0/2749283167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff714007820 con 0x7ff72009caf0 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff71d7fa640 1 -- 172.21.15.28:0/2749283167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff714007bf0 con 0x7ff72009caf0 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.145+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff72006da20 con 0x7ff72009caf0 2024-04-25T19:12:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.146+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff72006dfe0 con 0x7ff72009caf0 2024-04-25T19:12:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.147+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff720050740 con 0x7ff72009caf0 2024-04-25T19:12:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.147+0000 7ff71d7fa640 1 -- 172.21.15.28:0/2749283167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff714018070 con 0x7ff72009caf0 2024-04-25T19:12:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.147+0000 7ff71d7fa640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff7040744b0 0x7ff7040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.148+0000 7ff71d7fa640 1 -- 172.21.15.28:0/2749283167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff7140954c0 con 0x7ff72009caf0 2024-04-25T19:12:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.148+0000 7ff71ffff640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff7040744b0 0x7ff7040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.148+0000 7ff71ffff640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff7040744b0 0x7ff7040769c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff70c007fd0 tx=0x7ff70c007d80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.152+0000 7ff71d7fa640 1 -- 172.21.15.28:0/2749283167 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff71405f850 con 0x7ff72009caf0 2024-04-25T19:12:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.336+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff7201e9440 con 0x7ff72009caf0 2024-04-25T19:12:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.337+0000 7ff71d7fa640 1 -- 172.21.15.28:0/2749283167 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7ff71405f150 con 0x7ff72009caf0 2024-04-25T19:12:21.339 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff7040744b0 msgr2=0x7ff7040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff7040744b0 0x7ff7040769c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff70c007fd0 tx=0x7ff70c007d80 comp rx=0 tx=0).stop 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009caf0 msgr2=0x7ff72006eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009caf0 0x7ff72006eac0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff71400c7c0 tx=0x7ff71400cc80 comp rx=0 tx=0).stop 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 shutdown_connections 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff7040744b0 0x7ff7040769c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7200d2670 0x7ff7200c8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff72009cf00 0x7ff7201d3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 --2- 172.21.15.28:0/2749283167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff72009caf0 0x7ff72006eac0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 >> 172.21.15.28:0/2749283167 conn(0x7ff7200d03f0 msgr2=0x7ff720071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 shutdown_connections 2024-04-25T19:12:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.339+0000 7ff726a84640 1 -- 172.21.15.28:0/2749283167 wait complete. 2024-04-25T19:12:21.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:12:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.464+0000 7f1126061640 1 Processor -- start 2024-04-25T19:12:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.464+0000 7f1126061640 1 -- start start 2024-04-25T19:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f1126061640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f11200d22b0 0x7f11200cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f1126061640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f11200b7880 0x7f11200b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f1126061640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f112009d260 con 0x7f11200d1ea0 2024-04-25T19:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f1126061640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f112009caf0 con 0x7f11200d22b0 2024-04-25T19:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f1126061640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f11200d2890 con 0x7f11200b7880 2024-04-25T19:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111effd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f11200d22b0 0x7f11200cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111f7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f11200d1ea0 0x7f11200b7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36840/0 (socket says 172.21.15.28:36840) 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111effd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f11200d22b0 0x7f11200cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38130/0 (socket says 172.21.15.28:38130) 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111f7fe640 1 -- 172.21.15.28:0/2134509744 learned_addr learned my addr 172.21.15.28:0/2134509744 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111ffff640 1 --2- 172.21.15.28:0/2134509744 >> v2:172.21.15.53:3300/0 conn(0x7f11200b7880 0x7f11200b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111effd640 1 -- 172.21.15.28:0/2134509744 >> v2:172.21.15.53:3300/0 conn(0x7f11200b7880 msgr2=0x7f11200b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111effd640 1 --2- 172.21.15.28:0/2134509744 >> v2:172.21.15.53:3300/0 conn(0x7f11200b7880 0x7f11200b7c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111effd640 1 -- 172.21.15.28:0/2134509744 >> v1:172.21.15.28:6789/0 conn(0x7f11200d1ea0 legacy=0x7f11200b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111e7fc640 1 -- 172.21.15.28:0/2134509744 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2563590668 0 0) 0x7f112009d260 con 0x7f11200d1ea0 2024-04-25T19:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.466+0000 7f111effd640 1 -- 172.21.15.28:0/2134509744 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11200b9000 con 0x7f11200d22b0 2024-04-25T19:12:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.467+0000 7f111effd640 1 --2- 172.21.15.28:0/2134509744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11200cf640 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f11200d15e0 tx=0x7f111402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc0765b1633691f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.467+0000 7f111e7fc640 1 -- 172.21.15.28:0/2134509744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114037070 con 0x7f11200d22b0 2024-04-25T19:12:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.467+0000 7f111e7fc640 1 -- 172.21.15.28:0/2134509744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1114032440 con 0x7f11200d22b0 2024-04-25T19:12:21.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.467+0000 7f111e7fc640 1 -- 172.21.15.28:0/2134509744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111402ae60 con 0x7f11200d22b0 2024-04-25T19:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.468+0000 7f1126061640 1 -- 172.21.15.28:0/2134509744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 msgr2=0x7f11200cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.468+0000 7f1126061640 1 --2- 172.21.15.28:0/2134509744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11200cf640 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f11200d15e0 tx=0x7f111402a110 comp rx=0 tx=0).stop 2024-04-25T19:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.468+0000 7f1126061640 1 -- 172.21.15.28:0/2134509744 shutdown_connections 2024-04-25T19:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.468+0000 7f1126061640 1 --2- 172.21.15.28:0/2134509744 >> v2:172.21.15.53:3300/0 conn(0x7f11200b7880 0x7f11200b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.468+0000 7f1126061640 1 --2- 172.21.15.28:0/2134509744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11200cf640 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.468+0000 7f1126061640 1 -- 172.21.15.28:0/2134509744 >> 172.21.15.28:0/2134509744 conn(0x7f11200b5400 msgr2=0x7f11200b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.468+0000 7f1126061640 1 -- 172.21.15.28:0/2134509744 shutdown_connections 2024-04-25T19:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.468+0000 7f1126061640 1 -- 172.21.15.28:0/2134509744 wait complete. 2024-04-25T19:12:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.470+0000 7f1126061640 1 Processor -- start 2024-04-25T19:12:21.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.470+0000 7f1126061640 1 -- start start 2024-04-25T19:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f1126061640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11200b7880 0x7f11201f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f1126061640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11200d1ea0 0x7f11201d4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f1126061640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11201d5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f1126061640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11200d1770 con 0x7f11200d22b0 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f1126061640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f11200b81d0 con 0x7f11200d1ea0 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f1126061640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f11200d18f0 con 0x7f11200b7880 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f111ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11201d5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f111ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11201d5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38142/0 (socket says 172.21.15.28:38142) 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f111ffff640 1 -- 172.21.15.28:0/3337878163 learned_addr learned my addr 172.21.15.28:0/3337878163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f111effd640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11200d1ea0 0x7f11201d4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.471+0000 7f111f7fe640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11200b7880 0x7f11201f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f111ffff640 1 -- 172.21.15.28:0/3337878163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11200b7880 msgr2=0x7f11201f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f111ffff640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11200b7880 0x7f11201f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f111ffff640 1 -- 172.21.15.28:0/3337878163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11200d1ea0 msgr2=0x7f11201d4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f111ffff640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11200d1ea0 0x7f11201d4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f111ffff640 1 -- 172.21.15.28:0/3337878163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11201d5880 con 0x7f11200d22b0 2024-04-25T19:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f111ffff640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11201d5070 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f111800cc00 tx=0x7f111800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f1107fff640 1 -- 172.21.15.28:0/3337878163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111800ce10 con 0x7f11200d22b0 2024-04-25T19:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f1107fff640 1 -- 172.21.15.28:0/3337878163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1118003990 con 0x7f11200d22b0 2024-04-25T19:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f1107fff640 1 -- 172.21.15.28:0/3337878163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118003d60 con 0x7f11200d22b0 2024-04-25T19:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11201d5b60 con 0x7f11200d22b0 2024-04-25T19:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.472+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11201d6120 con 0x7f11200d22b0 2024-04-25T19:12:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.473+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1120072210 con 0x7f11200d22b0 2024-04-25T19:12:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.474+0000 7f1107fff640 1 -- 172.21.15.28:0/3337878163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1118003f40 con 0x7f11200d22b0 2024-04-25T19:12:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.474+0000 7f1107fff640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f110c0744b0 0x7f110c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.474+0000 7f1107fff640 1 -- 172.21.15.28:0/3337878163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f1118095b30 con 0x7f11200d22b0 2024-04-25T19:12:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.475+0000 7f111f7fe640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f110c0744b0 0x7f110c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.475+0000 7f111f7fe640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f110c0744b0 0x7f110c0769c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1110007fd0 tx=0x7f1110007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:21.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.479+0000 7f1107fff640 1 -- 172.21.15.28:0/3337878163 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1120072210 con 0x7f11200d22b0 2024-04-25T19:12:21.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.678+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f11201d55e0 con 0x7f11200d22b0 2024-04-25T19:12:21.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.679+0000 7f1107fff640 1 -- 172.21.15.28:0/3337878163 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f1118014030 con 0x7f11200d22b0 2024-04-25T19:12:21.681 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-04-25T19:12:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f110c0744b0 msgr2=0x7f110c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f110c0744b0 0x7f110c0769c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f1110007fd0 tx=0x7f1110007de0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 msgr2=0x7f11201d5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11201d5070 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f111800cc00 tx=0x7f111800a5c0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 shutdown_connections 2024-04-25T19:12:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f110c0744b0 0x7f110c0769c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11200d22b0 0x7f11201d5070 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11200d1ea0 0x7f11201d4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 --2- 172.21.15.28:0/3337878163 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11200b7880 0x7f11201f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:21.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 >> 172.21.15.28:0/3337878163 conn(0x7f11200b5400 msgr2=0x7f1120071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:21.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 shutdown_connections 2024-04-25T19:12:21.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:21.681+0000 7f1126061640 1 -- 172.21.15.28:0/3337878163 wait complete. 2024-04-25T19:12:21.694 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T19:12:21.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:21.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-25T19:12:21.720 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:12:21.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:21.721 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T19:12:21.776 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:12:21.776 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:21.796 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:12:21.796 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:21.856 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:12:21.856 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:21.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.027+0000 7fab8c9f3640 1 Processor -- start 2024-04-25T19:12:22.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.027+0000 7fab8c9f3640 1 -- start start 2024-04-25T19:12:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.028+0000 7fab8c9f3640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fab880b2960 0x7fab8809ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.028+0000 7fab8c9f3640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab880d1c90 con 0x7fab8808cc20 2024-04-25T19:12:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.028+0000 7fab8c9f3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab880cf730 con 0x7fab880b7210 2024-04-25T19:12:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.028+0000 7fab8c9f3640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fab880d28d0 con 0x7fab880b2960 2024-04-25T19:12:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.028+0000 7fab8659c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fab8808cc20 0x7fab880d21c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36850/0 (socket says 172.21.15.28:36850) 2024-04-25T19:12:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.028+0000 7fab8659c640 1 -- 172.21.15.28:0/1241568131 learned_addr learned my addr 172.21.15.28:0/1241568131 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457848380 0 0) 0x7fab78001290 con 0x7fab8808cc20 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab74003520 con 0x7fab8808cc20 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab86d9d640 1 --2- 172.21.15.28:0/1241568131 >> v2:172.21.15.53:3300/0 conn(0x7fab880b2960 0x7fab8809ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2621889582 0 0) 0x7fab74003520 con 0x7fab8808cc20 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 >> v2:172.21.15.53:3300/0 conn(0x7fab880b2960 msgr2=0x7fab8809ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 --2- 172.21.15.28:0/1241568131 >> v2:172.21.15.53:3300/0 conn(0x7fab880b2960 0x7fab8809ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 >> v1:172.21.15.53:6789/0 conn(0x7fab880b7210 legacy=0x7fab880d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab880cfa10 con 0x7fab8808cc20 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fab780019c0 con 0x7fab8808cc20 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 >> v1:172.21.15.28:6789/0 conn(0x7fab8808cc20 legacy=0x7fab880d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab740039b0 0x7fab74003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab74004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab74005250 0x7fab7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab880d28d0 con 0x7fab74004800 2024-04-25T19:12:22.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fab7400c1e0 con 0x7fab740039b0 2024-04-25T19:12:22.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fab7400c3e0 con 0x7fab74005250 2024-04-25T19:12:22.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab7dd9b640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab74004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab8659c640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab740039b0 0x7fab74003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.029+0000 7fab86d9d640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab74005250 0x7fab7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab7dd9b640 1 -- 172.21.15.28:0/1241568131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab74005250 msgr2=0x7fab7400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab7dd9b640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab74005250 0x7fab7400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab7dd9b640 1 -- 172.21.15.28:0/1241568131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab740039b0 msgr2=0x7fab74003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab7dd9b640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab740039b0 0x7fab74003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab7dd9b640 1 -- 172.21.15.28:0/1241568131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab74003520 con 0x7fab74004800 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab86d9d640 1 -- 172.21.15.28:0/1241568131 reap_dead start 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab7dd9b640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab74004c10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fab880cdfb0 tx=0x7fab74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cb86113a722f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70001370 con 0x7fab74004800 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab700052d0 con 0x7fab74004800 2024-04-25T19:12:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.030+0000 7fab85d9b640 1 -- 172.21.15.28:0/1241568131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7002ab90 con 0x7fab74004800 2024-04-25T19:12:22.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.031+0000 7fab8c9f3640 1 -- 172.21.15.28:0/1241568131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 msgr2=0x7fab74004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.031+0000 7fab8c9f3640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab74004c10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fab880cdfb0 tx=0x7fab74005830 comp rx=0 tx=0).stop 2024-04-25T19:12:22.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.031+0000 7fab8c9f3640 1 -- 172.21.15.28:0/1241568131 shutdown_connections 2024-04-25T19:12:22.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.031+0000 7fab8c9f3640 1 --2- 172.21.15.28:0/1241568131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab74004c10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.031+0000 7fab8c9f3640 1 -- 172.21.15.28:0/1241568131 >> 172.21.15.28:0/1241568131 conn(0x7fab880d0430 msgr2=0x7fab880d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.031+0000 7fab8c9f3640 1 -- 172.21.15.28:0/1241568131 shutdown_connections 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.031+0000 7fab8c9f3640 1 -- 172.21.15.28:0/1241568131 wait complete. 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.033+0000 7fab8c9f3640 1 Processor -- start 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.033+0000 7fab8c9f3640 1 -- start start 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8c9f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab8809ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8c9f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab8809e1c0 0x7fab8814e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8c9f3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab88150380 0x7fab88150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8c9f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab880b2d70 con 0x7fab74004800 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8c9f3640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fab880b8fb0 con 0x7fab88150380 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8c9f3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fab8809f620 con 0x7fab8809e1c0 2024-04-25T19:12:22.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab8809ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab8809ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38166/0 (socket says 172.21.15.28:38166) 2024-04-25T19:12:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 -- 172.21.15.28:0/2461715798 learned_addr learned my addr 172.21.15.28:0/2461715798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab86d9d640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab88150380 0x7fab88150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab7dd9b640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab8809e1c0 0x7fab8814e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 -- 172.21.15.28:0/2461715798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab8809e1c0 msgr2=0x7fab8814e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab8809e1c0 0x7fab8814e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 -- 172.21.15.28:0/2461715798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab88150380 msgr2=0x7fab88150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab88150380 0x7fab88150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 -- 172.21.15.28:0/2461715798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab8814f2e0 con 0x7fab74004800 2024-04-25T19:12:22.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8659c640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab8809ed90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fab78007270 tx=0x7fab78007730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab7f7fe640 1 -- 172.21.15.28:0/2461715798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab78017020 con 0x7fab74004800 2024-04-25T19:12:22.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab7f7fe640 1 -- 172.21.15.28:0/2461715798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab780013b0 con 0x7fab74004800 2024-04-25T19:12:22.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab8814f5c0 con 0x7fab74004800 2024-04-25T19:12:22.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab7f7fe640 1 -- 172.21.15.28:0/2461715798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7800fb60 con 0x7fab74004800 2024-04-25T19:12:22.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.035+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab8814fb80 con 0x7fab74004800 2024-04-25T19:12:22.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.036+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab88050740 con 0x7fab74004800 2024-04-25T19:12:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.037+0000 7fab7f7fe640 1 -- 172.21.15.28:0/2461715798 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fab7801a020 con 0x7fab74004800 2024-04-25T19:12:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.037+0000 7fab7f7fe640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fab680744b0 0x7fab680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.038+0000 7fab7dd9b640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fab680744b0 0x7fab680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.038+0000 7fab7f7fe640 1 -- 172.21.15.28:0/2461715798 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fab78095710 con 0x7fab74004800 2024-04-25T19:12:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.038+0000 7fab7dd9b640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fab680744b0 0x7fab680769c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fab70032000 tx=0x7fab7002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.042+0000 7fab7f7fe640 1 -- 172.21.15.28:0/2461715798 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab78021020 con 0x7fab74004800 2024-04-25T19:12:22.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.229+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab880b0a20 con 0x7fab74004800 2024-04-25T19:12:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.230+0000 7fab7f7fe640 1 -- 172.21.15.28:0/2461715798 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7fab880d1c90 con 0x7fab74004800 2024-04-25T19:12:22.233 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:22.233 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fab680744b0 msgr2=0x7fab680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fab680744b0 0x7fab680769c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fab70032000 tx=0x7fab7002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 msgr2=0x7fab8809ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab8809ed90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fab78007270 tx=0x7fab78007730 comp rx=0 tx=0).stop 2024-04-25T19:12:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 shutdown_connections 2024-04-25T19:12:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fab680744b0 0x7fab680769c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab88150380 0x7fab88150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fab8809e1c0 0x7fab8814e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 --2- 172.21.15.28:0/2461715798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab74004800 0x7fab8809ed90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 >> 172.21.15.28:0/2461715798 conn(0x7fab880d0430 msgr2=0x7fab88072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 shutdown_connections 2024-04-25T19:12:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.232+0000 7fab8c9f3640 1 -- 172.21.15.28:0/2461715798 wait complete. 2024-04-25T19:12:22.246 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:22.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.361+0000 7fb6e6346640 1 Processor -- start 2024-04-25T19:12:22.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.361+0000 7fb6e6346640 1 -- start start 2024-04-25T19:12:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6e6346640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb6e006edb0 0x7fb6e00b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6e6346640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6e01616a0 0x7fb6e009b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6e6346640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e00d1b30 0x7fb6e00d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6e6346640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e009c190 con 0x7fb6e01616a0 2024-04-25T19:12:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6e6346640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e0160c00 con 0x7fb6e006edb0 2024-04-25T19:12:22.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6e6346640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e009beb0 con 0x7fb6e00d1b30 2024-04-25T19:12:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6df7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6e01616a0 0x7fb6e009b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6df7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6e01616a0 0x7fb6e009b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38168/0 (socket says 172.21.15.28:38168) 2024-04-25T19:12:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6df7fe640 1 -- 172.21.15.28:0/628424906 learned_addr learned my addr 172.21.15.28:0/628424906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6e48e3640 1 --2- 172.21.15.28:0/628424906 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e00d1b30 0x7fb6e00d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.362+0000 7fb6dffff640 1 --2- 172.21.15.28:0/628424906 >> v2:172.21.15.53:3300/0 conn(0x7fb6e006edb0 0x7fb6e00b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.363+0000 7fb6df7fe640 1 -- 172.21.15.28:0/628424906 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e00d1b30 msgr2=0x7fb6e00d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.363+0000 7fb6df7fe640 1 --2- 172.21.15.28:0/628424906 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e00d1b30 0x7fb6e00d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.363+0000 7fb6df7fe640 1 -- 172.21.15.28:0/628424906 >> v2:172.21.15.53:3300/0 conn(0x7fb6e006edb0 msgr2=0x7fb6e00b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.363+0000 7fb6df7fe640 1 --2- 172.21.15.28:0/628424906 >> v2:172.21.15.53:3300/0 conn(0x7fb6e006edb0 0x7fb6e00b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.363+0000 7fb6df7fe640 1 -- 172.21.15.28:0/628424906 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e0160ea0 con 0x7fb6e01616a0 2024-04-25T19:12:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.363+0000 7fb6dffff640 1 --2- 172.21.15.28:0/628424906 >> v2:172.21.15.53:3300/0 conn(0x7fb6e006edb0 0x7fb6e00b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.363+0000 7fb6df7fe640 1 --2- 172.21.15.28:0/628424906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e01616a0 0x7fb6e009b540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009870 tx=0x7fb6d402a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49adf05ece4928fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.363+0000 7fb6deffd640 1 -- 172.21.15.28:0/628424906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4037070 con 0x7fb6e01616a0 2024-04-25T19:12:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6deffd640 1 -- 172.21.15.28:0/628424906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d4032440 con 0x7fb6e01616a0 2024-04-25T19:12:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6deffd640 1 -- 172.21.15.28:0/628424906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4032810 con 0x7fb6e01616a0 2024-04-25T19:12:22.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 -- 172.21.15.28:0/628424906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e01616a0 msgr2=0x7fb6e009b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 --2- 172.21.15.28:0/628424906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e01616a0 0x7fb6e009b540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009870 tx=0x7fb6d402a200 comp rx=0 tx=0).stop 2024-04-25T19:12:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 -- 172.21.15.28:0/628424906 shutdown_connections 2024-04-25T19:12:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 --2- 172.21.15.28:0/628424906 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e00d1b30 0x7fb6e00d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 --2- 172.21.15.28:0/628424906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e01616a0 0x7fb6e009b540 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 --2- 172.21.15.28:0/628424906 >> v2:172.21.15.53:3300/0 conn(0x7fb6e006edb0 0x7fb6e00b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 -- 172.21.15.28:0/628424906 >> 172.21.15.28:0/628424906 conn(0x7fb6e00b7620 msgr2=0x7fb6e00b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 -- 172.21.15.28:0/628424906 shutdown_connections 2024-04-25T19:12:22.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.364+0000 7fb6e6346640 1 -- 172.21.15.28:0/628424906 wait complete. 2024-04-25T19:12:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.366+0000 7fb6e6346640 1 Processor -- start 2024-04-25T19:12:22.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.366+0000 7fb6e6346640 1 -- start start 2024-04-25T19:12:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.367+0000 7fb6e6346640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6e006edb0 0x7fb6e01573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.367+0000 7fb6e6346640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e00d1b30 0x7fb6e0162d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.367+0000 7fb6e6346640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e01616a0 0x7fb6e014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.367+0000 7fb6e6346640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e00d2480 con 0x7fb6e00d1b30 2024-04-25T19:12:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6e6346640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e00b4180 con 0x7fb6e006edb0 2024-04-25T19:12:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6e6346640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e009ba80 con 0x7fb6e01616a0 2024-04-25T19:12:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e00d1b30 0x7fb6e0162d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e00d1b30 0x7fb6e0162d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38174/0 (socket says 172.21.15.28:38174) 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 -- 172.21.15.28:0/1660145678 learned_addr learned my addr 172.21.15.28:0/1660145678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6e48e3640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e01616a0 0x7fb6e014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6dffff640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6e006edb0 0x7fb6e01573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 -- 172.21.15.28:0/1660145678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e01616a0 msgr2=0x7fb6e014f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e01616a0 0x7fb6e014f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 -- 172.21.15.28:0/1660145678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6e006edb0 msgr2=0x7fb6e01573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6e006edb0 0x7fb6e01573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 -- 172.21.15.28:0/1660145678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e0157ba0 con 0x7fb6e00d1b30 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6df7fe640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e00d1b30 0x7fb6e0162d70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4000c00 tx=0x7fb6d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6dcff9640 1 -- 172.21.15.28:0/1660145678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4041020 con 0x7fb6e00d1b30 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6dcff9640 1 -- 172.21.15.28:0/1660145678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d40306c0 con 0x7fb6e00d1b30 2024-04-25T19:12:22.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.368+0000 7fb6dcff9640 1 -- 172.21.15.28:0/1660145678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4037040 con 0x7fb6e00d1b30 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.369+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e0157e80 con 0x7fb6e00d1b30 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.369+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e0158440 con 0x7fb6e00d1b30 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.369+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6e0158680 con 0x7fb6e00d1b30 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.370+0000 7fb6dcff9640 1 -- 172.21.15.28:0/1660145678 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb6d4044020 con 0x7fb6e00d1b30 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.371+0000 7fb6dcff9640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6cc0744b0 0x7fb6cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.371+0000 7fb6dcff9640 1 -- 172.21.15.28:0/1660145678 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb6d40b5d00 con 0x7fb6e00d1b30 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.371+0000 7fb6dffff640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6cc0744b0 0x7fb6cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.371+0000 7fb6dffff640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6cc0744b0 0x7fb6cc0769c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb6e006e970 tx=0x7fb6d0004730 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.375+0000 7fb6dcff9640 1 -- 172.21.15.28:0/1660145678 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6d404b030 con 0x7fb6e00d1b30 2024-04-25T19:12:22.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.560+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb6e0157930 con 0x7fb6e00d1b30 2024-04-25T19:12:22.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.561+0000 7fb6dcff9640 1 -- 172.21.15.28:0/1660145678 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fb6d407fa40 con 0x7fb6e00d1b30 2024-04-25T19:12:22.564 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.562+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6cc0744b0 msgr2=0x7fb6cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.562+0000 7fb6e6346640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6cc0744b0 0x7fb6cc0769c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb6e006e970 tx=0x7fb6d0004730 comp rx=0 tx=0).stop 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.562+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e00d1b30 msgr2=0x7fb6e0162d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.562+0000 7fb6e6346640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e00d1b30 0x7fb6e0162d70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4000c00 tx=0x7fb6d40046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.562+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 shutdown_connections 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.563+0000 7fb6e6346640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6cc0744b0 0x7fb6cc0769c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.563+0000 7fb6e6346640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6e01616a0 0x7fb6e014f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.563+0000 7fb6e6346640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6e00d1b30 0x7fb6e0162d70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.563+0000 7fb6e6346640 1 --2- 172.21.15.28:0/1660145678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6e006edb0 0x7fb6e01573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.563+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 >> 172.21.15.28:0/1660145678 conn(0x7fb6e00b7620 msgr2=0x7fb6e0163f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.563+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 shutdown_connections 2024-04-25T19:12:22.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.563+0000 7fb6e6346640 1 -- 172.21.15.28:0/1660145678 wait complete. 2024-04-25T19:12:22.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.689+0000 7ff4631b7640 1 Processor -- start 2024-04-25T19:12:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.689+0000 7ff4631b7640 1 -- start start 2024-04-25T19:12:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff4631b7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff45c15f270 0x7ff45c0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff4631b7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c0b49e0 0x7ff45c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff4631b7640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff45c0b4df0 con 0x7ff45c0cead0 2024-04-25T19:12:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff4631b7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7ff45c15ede0 con 0x7ff45c15f270 2024-04-25T19:12:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff4631b7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff45c15e540 con 0x7ff45c0b49e0 2024-04-25T19:12:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff460f53640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff45c15f270 0x7ff45c0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff460f53640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff45c15f270 0x7ff45c0b44a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38182/0 (socket says 172.21.15.28:38182) 2024-04-25T19:12:22.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff460f53640 1 -- 172.21.15.28:0/3876433414 learned_addr learned my addr 172.21.15.28:0/3876433414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff45bfff640 1 --2- 172.21.15.28:0/3876433414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c0b49e0 0x7ff45c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff460f53640 1 -- 172.21.15.28:0/3876433414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c0b49e0 msgr2=0x7ff45c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff460f53640 1 --2- 172.21.15.28:0/3876433414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c0b49e0 0x7ff45c0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff460f53640 1 -- 172.21.15.28:0/3876433414 >> v1:172.21.15.53:6789/0 conn(0x7ff45c0cead0 legacy=0x7ff45c0b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.691+0000 7ff460f53640 1 -- 172.21.15.28:0/3876433414 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c15f680 con 0x7ff45c15f270 2024-04-25T19:12:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.692+0000 7ff460f53640 1 --2- 172.21.15.28:0/3876433414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c15f270 0x7ff45c0b44a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009870 tx=0x7ff44c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21fac7b3a6af3a8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.692+0000 7ff45b7fe640 1 -- 172.21.15.28:0/3876433414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c037070 con 0x7ff45c15f270 2024-04-25T19:12:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.692+0000 7ff45b7fe640 1 -- 172.21.15.28:0/3876433414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44c032440 con 0x7ff45c15f270 2024-04-25T19:12:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.692+0000 7ff45b7fe640 1 -- 172.21.15.28:0/3876433414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c032810 con 0x7ff45c15f270 2024-04-25T19:12:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.693+0000 7ff4631b7640 1 -- 172.21.15.28:0/3876433414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c15f270 msgr2=0x7ff45c0b44a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.693+0000 7ff4631b7640 1 --2- 172.21.15.28:0/3876433414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c15f270 0x7ff45c0b44a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009870 tx=0x7ff44c02a210 comp rx=0 tx=0).stop 2024-04-25T19:12:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.693+0000 7ff4631b7640 1 -- 172.21.15.28:0/3876433414 shutdown_connections 2024-04-25T19:12:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.693+0000 7ff4631b7640 1 --2- 172.21.15.28:0/3876433414 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c0b49e0 0x7ff45c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.693+0000 7ff4631b7640 1 --2- 172.21.15.28:0/3876433414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c15f270 0x7ff45c0b44a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.693+0000 7ff4631b7640 1 -- 172.21.15.28:0/3876433414 >> 172.21.15.28:0/3876433414 conn(0x7ff45c0b7640 msgr2=0x7ff45c0b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.693+0000 7ff4631b7640 1 -- 172.21.15.28:0/3876433414 shutdown_connections 2024-04-25T19:12:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.693+0000 7ff4631b7640 1 -- 172.21.15.28:0/3876433414 wait complete. 2024-04-25T19:12:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.694+0000 7ff4631b7640 1 Processor -- start 2024-04-25T19:12:22.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.695+0000 7ff4631b7640 1 -- start start 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff4631b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c0b49e0 0x7ff45c154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff4631b7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff45c0cead0 0x7ff45c0c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff4631b7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c15f270 0x7ff45c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff4631b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c15ea70 con 0x7ff45c0b49e0 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff4631b7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff45c15e790 con 0x7ff45c0cead0 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff4631b7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff45c0ceee0 con 0x7ff45c15f270 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff460f53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c0b49e0 0x7ff45c154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff460f53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c0b49e0 0x7ff45c154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38198/0 (socket says 172.21.15.28:38198) 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff460f53640 1 -- 172.21.15.28:0/1044206761 learned_addr learned my addr 172.21.15.28:0/1044206761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff460f53640 1 -- 172.21.15.28:0/1044206761 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c15f270 msgr2=0x7ff45c0c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff460f53640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c15f270 0x7ff45c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff45bfff640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff45c0cead0 0x7ff45c0c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff460f53640 1 -- 172.21.15.28:0/1044206761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff45c0cead0 msgr2=0x7ff45c0c8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff460f53640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff45c0cead0 0x7ff45c0c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.696+0000 7ff460f53640 1 -- 172.21.15.28:0/1044206761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff45c0c67e0 con 0x7ff45c0b49e0 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.697+0000 7ff460f53640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c0b49e0 0x7ff45c154d90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009870 tx=0x7ff44c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.697+0000 7ff4597fa640 1 -- 172.21.15.28:0/1044206761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c041020 con 0x7ff45c0b49e0 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.697+0000 7ff4597fa640 1 -- 172.21.15.28:0/1044206761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff44c030970 con 0x7ff45c0b49e0 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.697+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff45c0c6ac0 con 0x7ff45c0b49e0 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.697+0000 7ff4597fa640 1 -- 172.21.15.28:0/1044206761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff44c037040 con 0x7ff45c0b49e0 2024-04-25T19:12:22.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.697+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff45c0c7080 con 0x7ff45c0b49e0 2024-04-25T19:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.698+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff45c050740 con 0x7ff45c0b49e0 2024-04-25T19:12:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.699+0000 7ff4597fa640 1 -- 172.21.15.28:0/1044206761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff44c044020 con 0x7ff45c0b49e0 2024-04-25T19:12:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.699+0000 7ff4597fa640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff4480744b0 0x7ff4480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.699+0000 7ff4597fa640 1 -- 172.21.15.28:0/1044206761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff44c0b60d0 con 0x7ff45c0b49e0 2024-04-25T19:12:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.699+0000 7ff45bfff640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff4480744b0 0x7ff4480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.700+0000 7ff45bfff640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff4480744b0 0x7ff4480769c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff45c1f84b0 tx=0x7ff450008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:22.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.704+0000 7ff4597fa640 1 -- 172.21.15.28:0/1044206761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff44c04b030 con 0x7ff45c0b49e0 2024-04-25T19:12:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.888+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff45c069a80 con 0x7ff45c0b49e0 2024-04-25T19:12:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.888+0000 7ff4597fa640 1 -- 172.21.15.28:0/1044206761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7ff45c0b4df0 con 0x7ff45c0b49e0 2024-04-25T19:12:22.891 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff4480744b0 msgr2=0x7ff4480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff4480744b0 0x7ff4480769c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff45c1f84b0 tx=0x7ff450008040 comp rx=0 tx=0).stop 2024-04-25T19:12:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c0b49e0 msgr2=0x7ff45c154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c0b49e0 0x7ff45c154d90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff44c009870 tx=0x7ff44c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 shutdown_connections 2024-04-25T19:12:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff4480744b0 0x7ff4480769c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff45c15f270 0x7ff45c0c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff45c0cead0 0x7ff45c0c8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 --2- 172.21.15.28:0/1044206761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff45c0b49e0 0x7ff45c154d90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 >> 172.21.15.28:0/1044206761 conn(0x7ff45c0b7640 msgr2=0x7ff45c0b2310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:22.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 shutdown_connections 2024-04-25T19:12:22.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:22.890+0000 7ff4631b7640 1 -- 172.21.15.28:0/1044206761 wait complete. 2024-04-25T19:12:22.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.017+0000 7fd3716aa640 1 Processor -- start 2024-04-25T19:12:23.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.017+0000 7fd3716aa640 1 -- start start 2024-04-25T19:12:23.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.018+0000 7fd3716aa640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd36c160310 0x7fd36c0d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.018+0000 7fd3716aa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c0b79f0 0x7fd36c0b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.018+0000 7fd3716aa640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd36c160720 con 0x7fd36c08cce0 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.018+0000 7fd3716aa640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fd36c0b8340 con 0x7fd36c160310 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.018+0000 7fd3716aa640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd36c0d5640 con 0x7fd36c0b79f0 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd36b7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd36c08cce0 0x7fd36c08d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36864/0 (socket says 172.21.15.28:36864) 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd36b7fe640 1 -- 172.21.15.28:0/4096837608 learned_addr learned my addr 172.21.15.28:0/4096837608 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd36affd640 1 --2- 172.21.15.28:0/4096837608 >> v2:172.21.15.53:3300/0 conn(0x7fd36c160310 0x7fd36c0d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd36a7fc640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c0b79f0 0x7fd36c0b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 511715227 0 0) 0x7fd36c160720 con 0x7fd36c08cce0 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd358003520 con 0x7fd36c08cce0 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3565188711 0 0) 0x7fd358003520 con 0x7fd36c08cce0 2024-04-25T19:12:23.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c0b79f0 msgr2=0x7fd36c0b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c0b79f0 0x7fd36c0b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 >> v2:172.21.15.53:3300/0 conn(0x7fd36c160310 msgr2=0x7fd36c0d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 --2- 172.21.15.28:0/4096837608 >> v2:172.21.15.53:3300/0 conn(0x7fd36c160310 0x7fd36c0d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36c0d0f40 con 0x7fd36c08cce0 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fd35c0019f0 con 0x7fd36c08cce0 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 >> v1:172.21.15.28:6789/0 conn(0x7fd36c08cce0 legacy=0x7fd36c08d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd358003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd358004800 0x7fd358004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c0b79f0 0x7fd358005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c0b8340 con 0x7fd3580039b0 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c0d5640 con 0x7fd358004800 2024-04-25T19:12:23.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd35800b9f0 con 0x7fd36c0b79f0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.019+0000 7fd36affd640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd358003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.020+0000 7fd36a7fc640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd358004800 0x7fd358004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.020+0000 7fd36b7fe640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c0b79f0 0x7fd358005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.020+0000 7fd36affd640 1 -- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c0b79f0 msgr2=0x7fd358005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.020+0000 7fd36affd640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c0b79f0 0x7fd358005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.020+0000 7fd36affd640 1 -- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd358004800 msgr2=0x7fd358004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.020+0000 7fd36affd640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd358004800 0x7fd358004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.020+0000 7fd36affd640 1 -- 172.21.15.28:0/4096837608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd358003520 con 0x7fd3580039b0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.020+0000 7fd36affd640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd358003e20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ac0 tx=0x7fd358005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5adf77fc9f6073 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364036070 con 0x7fd3580039b0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd364031400 con 0x7fd3580039b0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd369ffb640 1 -- 172.21.15.28:0/4096837608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364009690 con 0x7fd3580039b0 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd3716aa640 1 -- 172.21.15.28:0/4096837608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 msgr2=0x7fd358003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd3716aa640 1 --2- 172.21.15.28:0/4096837608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd358003e20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ac0 tx=0x7fd358005830 comp rx=0 tx=0).stop 2024-04-25T19:12:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd36b7fe640 1 -- 172.21.15.28:0/4096837608 reap_dead start 2024-04-25T19:12:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd3716aa640 1 -- 172.21.15.28:0/4096837608 shutdown_connections 2024-04-25T19:12:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd3716aa640 1 -- 172.21.15.28:0/4096837608 >> 172.21.15.28:0/4096837608 conn(0x7fd36c0d2a50 msgr2=0x7fd36c0d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd3716aa640 1 -- 172.21.15.28:0/4096837608 shutdown_connections 2024-04-25T19:12:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.021+0000 7fd3716aa640 1 -- 172.21.15.28:0/4096837608 wait complete. 2024-04-25T19:12:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.023+0000 7fd3716aa640 1 Processor -- start 2024-04-25T19:12:23.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.023+0000 7fd3716aa640 1 -- start start 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3716aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd36c1f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3716aa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd36c156360 0x7fd36c147440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3716aa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c147980 0x7fd36c1491d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3716aa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c0b9400 con 0x7fd3580039b0 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3716aa640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd36c0d1500 con 0x7fd36c156360 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3716aa640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd36c0d0bf0 con 0x7fd36c147980 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd36c1f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd36c1f61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38206/0 (socket says 172.21.15.28:38206) 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 -- 172.21.15.28:0/3235728021 learned_addr learned my addr 172.21.15.28:0/3235728021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36a7fc640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd36c156360 0x7fd36c147440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36b7fe640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c147980 0x7fd36c1491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 -- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c147980 msgr2=0x7fd36c1491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c147980 0x7fd36c1491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 -- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd36c156360 msgr2=0x7fd36c147440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd36c156360 0x7fd36c147440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 -- 172.21.15.28:0/3235728021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36c148130 con 0x7fd3580039b0 2024-04-25T19:12:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36b7fe640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c147980 0x7fd36c1491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd36affd640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd36c1f61b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ac0 tx=0x7fd364004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3637fe640 1 -- 172.21.15.28:0/3235728021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364041020 con 0x7fd3580039b0 2024-04-25T19:12:23.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3637fe640 1 -- 172.21.15.28:0/3235728021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd36403c470 con 0x7fd3580039b0 2024-04-25T19:12:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3637fe640 1 -- 172.21.15.28:0/3235728021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd364036040 con 0x7fd3580039b0 2024-04-25T19:12:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.025+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36c148410 con 0x7fd3580039b0 2024-04-25T19:12:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.026+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd36c1489d0 con 0x7fd3580039b0 2024-04-25T19:12:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.027+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd36c050740 con 0x7fd3580039b0 2024-04-25T19:12:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.027+0000 7fd3637fe640 1 -- 172.21.15.28:0/3235728021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd364044020 con 0x7fd3580039b0 2024-04-25T19:12:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.027+0000 7fd3637fe640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd34c0744b0 0x7fd34c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.028+0000 7fd3637fe640 1 -- 172.21.15.28:0/3235728021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd3640b5ee0 con 0x7fd3580039b0 2024-04-25T19:12:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.028+0000 7fd36a7fc640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd34c0744b0 0x7fd34c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.028+0000 7fd36a7fc640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd34c0744b0 0x7fd34c0769c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd36c0b55b0 tx=0x7fd354008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.032+0000 7fd3637fe640 1 -- 172.21.15.28:0/3235728021 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd36404b030 con 0x7fd3580039b0 2024-04-25T19:12:23.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.218+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd36c15fec0 con 0x7fd3580039b0 2024-04-25T19:12:23.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.218+0000 7fd3637fe640 1 -- 172.21.15.28:0/3235728021 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fd364080270 con 0x7fd3580039b0 2024-04-25T19:12:23.222 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd34c0744b0 msgr2=0x7fd34c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd34c0744b0 0x7fd34c0769c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd36c0b55b0 tx=0x7fd354008040 comp rx=0 tx=0).stop 2024-04-25T19:12:23.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 msgr2=0x7fd36c1f61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd36c1f61b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ac0 tx=0x7fd364004730 comp rx=0 tx=0).stop 2024-04-25T19:12:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 shutdown_connections 2024-04-25T19:12:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd34c0744b0 0x7fd34c0769c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd36c147980 0x7fd36c1491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd36c156360 0x7fd36c147440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 --2- 172.21.15.28:0/3235728021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3580039b0 0x7fd36c1f61b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 >> 172.21.15.28:0/3235728021 conn(0x7fd36c0d2a50 msgr2=0x7fd36c0b6830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 shutdown_connections 2024-04-25T19:12:23.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.220+0000 7fd3716aa640 1 -- 172.21.15.28:0/3235728021 wait complete. 2024-04-25T19:12:23.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.347+0000 7f5ce4ef9640 1 Processor -- start 2024-04-25T19:12:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.347+0000 7f5ce4ef9640 1 -- start start 2024-04-25T19:12:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5ce4ef9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ce00b81a0 0x7f5ce00d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5ce4ef9640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce00d2ee0 con 0x7f5ce0160250 2024-04-25T19:12:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5ce4ef9640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce0164b20 con 0x7f5ce00d2ad0 2024-04-25T19:12:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5ce4ef9640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f5ce00b4f50 con 0x7f5ce00b81a0 2024-04-25T19:12:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5cddd9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ce00b81a0 0x7f5ce00d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5cde59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ce0160250 0x7f5ce00b4690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36868/0 (socket says 172.21.15.28:36868) 2024-04-25T19:12:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5cde59c640 1 -- 172.21.15.28:0/2920529494 learned_addr learned my addr 172.21.15.28:0/2920529494 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5cddd9b640 1 -- 172.21.15.28:0/2920529494 >> v1:172.21.15.53:6789/0 conn(0x7f5ce00d2ad0 legacy=0x7f5ce00d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5cddd9b640 1 -- 172.21.15.28:0/2920529494 >> v1:172.21.15.28:6789/0 conn(0x7f5ce0160250 legacy=0x7f5ce00b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5cdd59a640 1 -- 172.21.15.28:0/2920529494 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3163631002 0 0) 0x7f5cd8001110 con 0x7f5ce0160250 2024-04-25T19:12:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.349+0000 7f5cddd9b640 1 -- 172.21.15.28:0/2920529494 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce0163d60 con 0x7f5ce00b81a0 2024-04-25T19:12:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.350+0000 7f5cddd9b640 1 --2- 172.21.15.28:0/2920529494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce00d2590 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097a0 tx=0x7f5cd402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7365cb9272c312f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.350+0000 7f5cdd59a640 1 -- 172.21.15.28:0/2920529494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4037070 con 0x7f5ce00b81a0 2024-04-25T19:12:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.350+0000 7f5cdd59a640 1 -- 172.21.15.28:0/2920529494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd402ae50 con 0x7f5ce00b81a0 2024-04-25T19:12:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.350+0000 7f5cdd59a640 1 -- 172.21.15.28:0/2920529494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4032660 con 0x7f5ce00b81a0 2024-04-25T19:12:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.351+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/2920529494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 msgr2=0x7f5ce00d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.351+0000 7f5ce4ef9640 1 --2- 172.21.15.28:0/2920529494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce00d2590 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5cd40097a0 tx=0x7f5cd402a140 comp rx=0 tx=0).stop 2024-04-25T19:12:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.351+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/2920529494 shutdown_connections 2024-04-25T19:12:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.351+0000 7f5ce4ef9640 1 --2- 172.21.15.28:0/2920529494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce00d2590 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.351+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/2920529494 >> 172.21.15.28:0/2920529494 conn(0x7f5ce006ee30 msgr2=0x7f5ce006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.351+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/2920529494 shutdown_connections 2024-04-25T19:12:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.351+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/2920529494 wait complete. 2024-04-25T19:12:23.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.353+0000 7f5ce4ef9640 1 Processor -- start 2024-04-25T19:12:23.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.353+0000 7f5ce4ef9640 1 -- start start 2024-04-25T19:12:23.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.354+0000 7f5ce4ef9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce015c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.354+0000 7f5ce4ef9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ce00d2ad0 0x7f5ce015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.354+0000 7f5ce4ef9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ce0160250 0x7f5ce0162880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.354+0000 7f5ce4ef9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce0164600 con 0x7f5ce00b81a0 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.354+0000 7f5ce4ef9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce0164320 con 0x7f5ce00d2ad0 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.354+0000 7f5ce4ef9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce00d4bc0 con 0x7f5ce0160250 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce015c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce015c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38232/0 (socket says 172.21.15.28:38232) 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 -- 172.21.15.28:0/3780291559 learned_addr learned my addr 172.21.15.28:0/3780291559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cddd9b640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ce00d2ad0 0x7f5ce015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cded9d640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ce0160250 0x7f5ce0162880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 -- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ce0160250 msgr2=0x7f5ce0162880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ce0160250 0x7f5ce0162880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 -- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ce00d2ad0 msgr2=0x7f5ce015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ce00d2ad0 0x7f5ce015bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 -- 172.21.15.28:0/3780291559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce015cca0 con 0x7f5ce00b81a0 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cddd9b640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ce00d2ad0 0x7f5ce015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cde59c640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce015c3c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5cd800c440 tx=0x7f5cd800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cceffd640 1 -- 172.21.15.28:0/3780291559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8018070 con 0x7f5ce00b81a0 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cceffd640 1 -- 172.21.15.28:0/3780291559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cd80074e0 con 0x7f5ce00b81a0 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5cceffd640 1 -- 172.21.15.28:0/3780291559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8004a20 con 0x7f5ce00b81a0 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce015cf80 con 0x7f5ce00b81a0 2024-04-25T19:12:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.355+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce015d540 con 0x7f5ce00b81a0 2024-04-25T19:12:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.356+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce00d45b0 con 0x7f5ce00b81a0 2024-04-25T19:12:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.357+0000 7f5cceffd640 1 -- 172.21.15.28:0/3780291559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5cd801b070 con 0x7f5ce00b81a0 2024-04-25T19:12:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.357+0000 7f5cceffd640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cc00744e0 0x7f5cc00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.357+0000 7f5cddd9b640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cc00744e0 0x7f5cc00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.357+0000 7f5cceffd640 1 -- 172.21.15.28:0/3780291559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5cd8094940 con 0x7f5ce00b81a0 2024-04-25T19:12:23.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.358+0000 7f5cddd9b640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cc00744e0 0x7f5cc00769f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4034000 tx=0x7f5cd402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.362+0000 7f5cceffd640 1 -- 172.21.15.28:0/3780291559 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5ce00d45b0 con 0x7f5ce00b81a0 2024-04-25T19:12:23.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.547+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5ce0050740 con 0x7f5ce00b81a0 2024-04-25T19:12:23.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.548+0000 7f5cceffd640 1 -- 172.21.15.28:0/3780291559 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f5ce00d2ee0 con 0x7f5ce00b81a0 2024-04-25T19:12:23.551 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cc00744e0 msgr2=0x7f5cc00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cc00744e0 0x7f5cc00769f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4034000 tx=0x7f5cd402e000 comp rx=0 tx=0).stop 2024-04-25T19:12:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 msgr2=0x7f5ce015c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce015c3c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5cd800c440 tx=0x7f5cd800c900 comp rx=0 tx=0).stop 2024-04-25T19:12:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 shutdown_connections 2024-04-25T19:12:23.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5cc00744e0 0x7f5cc00769f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5ce0160250 0x7f5ce0162880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ce00d2ad0 0x7f5ce015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 --2- 172.21.15.28:0/3780291559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce00b81a0 0x7f5ce015c3c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 >> 172.21.15.28:0/3780291559 conn(0x7f5ce006ee30 msgr2=0x7f5ce00d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 shutdown_connections 2024-04-25T19:12:23.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.550+0000 7f5ce4ef9640 1 -- 172.21.15.28:0/3780291559 wait complete. 2024-04-25T19:12:23.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:12:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.674+0000 7f448960c640 1 Processor -- start 2024-04-25T19:12:23.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.674+0000 7f448960c640 1 -- start start 2024-04-25T19:12:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f448960c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4484160310 0x7f44840d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f448960c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44840b9920 0x7f44840b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f448960c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4484160720 con 0x7f44840b79f0 2024-04-25T19:12:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f448960c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f44840d5640 con 0x7f44840b9920 2024-04-25T19:12:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f448960c640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f44840d0f40 con 0x7f4484160310 2024-04-25T19:12:23.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f44837fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44840b9920 0x7f44840b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f44837fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f44840b9920 0x7f44840b9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38234/0 (socket says 172.21.15.28:38234) 2024-04-25T19:12:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f44837fe640 1 -- 172.21.15.28:0/1479128991 learned_addr learned my addr 172.21.15.28:0/1479128991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f4482ffd640 1 --2- 172.21.15.28:0/1479128991 >> v2:172.21.15.53:3300/0 conn(0x7f4484160310 0x7f44840d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f44837fe640 1 -- 172.21.15.28:0/1479128991 >> v2:172.21.15.53:3300/0 conn(0x7f4484160310 msgr2=0x7f44840d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f44837fe640 1 --2- 172.21.15.28:0/1479128991 >> v2:172.21.15.53:3300/0 conn(0x7f4484160310 0x7f44840d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f44837fe640 1 -- 172.21.15.28:0/1479128991 >> v1:172.21.15.53:6789/0 conn(0x7f44840b79f0 legacy=0x7f44840b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.676+0000 7f44837fe640 1 -- 172.21.15.28:0/1479128991 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44840d1220 con 0x7f44840b9920 2024-04-25T19:12:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.677+0000 7f44837fe640 1 --2- 172.21.15.28:0/1479128991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840b9d30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f44840d6be0 tx=0x7f447402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e4a1ae6ad316317 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.677+0000 7f4481ffb640 1 -- 172.21.15.28:0/1479128991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4474036070 con 0x7f44840b9920 2024-04-25T19:12:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.677+0000 7f4481ffb640 1 -- 172.21.15.28:0/1479128991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4474031600 con 0x7f44840b9920 2024-04-25T19:12:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.677+0000 7f4481ffb640 1 -- 172.21.15.28:0/1479128991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44740319d0 con 0x7f44840b9920 2024-04-25T19:12:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.678+0000 7f448960c640 1 -- 172.21.15.28:0/1479128991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 msgr2=0x7f44840b9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.678+0000 7f448960c640 1 --2- 172.21.15.28:0/1479128991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840b9d30 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f44840d6be0 tx=0x7f447402a730 comp rx=0 tx=0).stop 2024-04-25T19:12:23.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.678+0000 7f448960c640 1 -- 172.21.15.28:0/1479128991 shutdown_connections 2024-04-25T19:12:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.678+0000 7f448960c640 1 --2- 172.21.15.28:0/1479128991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840b9d30 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.678+0000 7f448960c640 1 --2- 172.21.15.28:0/1479128991 >> v2:172.21.15.53:3300/0 conn(0x7f4484160310 0x7f44840d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.678+0000 7f448960c640 1 -- 172.21.15.28:0/1479128991 >> 172.21.15.28:0/1479128991 conn(0x7f44840d2a50 msgr2=0x7f44840d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.678+0000 7f448960c640 1 -- 172.21.15.28:0/1479128991 shutdown_connections 2024-04-25T19:12:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.678+0000 7f448960c640 1 -- 172.21.15.28:0/1479128991 wait complete. 2024-04-25T19:12:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.681+0000 7f448960c640 1 Processor -- start 2024-04-25T19:12:23.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.681+0000 7f448960c640 1 -- start start 2024-04-25T19:12:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f448960c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44840b79f0 0x7f448414b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f448960c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f448960c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4484160310 0x7f44840cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f448960c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f44840d0bf0 con 0x7f44840b9920 2024-04-25T19:12:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f448960c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f44840b9400 con 0x7f4484160310 2024-04-25T19:12:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f448960c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f44840cc950 con 0x7f44840b79f0 2024-04-25T19:12:23.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840cd700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38244/0 (socket says 172.21.15.28:38244) 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 -- 172.21.15.28:0/599013660 learned_addr learned my addr 172.21.15.28:0/599013660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f4482ffd640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44840b79f0 0x7f448414b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44837fe640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4484160310 0x7f44840cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 -- 172.21.15.28:0/599013660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44840b79f0 msgr2=0x7f448414b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44840b79f0 0x7f448414b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 -- 172.21.15.28:0/599013660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4484160310 msgr2=0x7f44840cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4484160310 0x7f44840cf180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 -- 172.21.15.28:0/599013660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44840ce0e0 con 0x7f44840b9920 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f44827fc640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840cd700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f446c00c470 tx=0x7f446c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f447b7fe640 1 -- 172.21.15.28:0/599013660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c0052d0 con 0x7f44840b9920 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f447b7fe640 1 -- 172.21.15.28:0/599013660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f446c005490 con 0x7f44840b9920 2024-04-25T19:12:23.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f447b7fe640 1 -- 172.21.15.28:0/599013660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f446c007260 con 0x7f44840b9920 2024-04-25T19:12:23.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44840ce3c0 con 0x7f44840b9920 2024-04-25T19:12:23.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.682+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44840ce980 con 0x7f44840b9920 2024-04-25T19:12:23.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.683+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f448408e430 con 0x7f44840b9920 2024-04-25T19:12:23.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.684+0000 7f447b7fe640 1 -- 172.21.15.28:0/599013660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f446c007400 con 0x7f44840b9920 2024-04-25T19:12:23.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.684+0000 7f447b7fe640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f44640744e0 0x7f44640769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:23.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.684+0000 7f447b7fe640 1 -- 172.21.15.28:0/599013660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f446c094a00 con 0x7f44840b9920 2024-04-25T19:12:23.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.684+0000 7f4482ffd640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f44640744e0 0x7f44640769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:23.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.685+0000 7f4482ffd640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f44640744e0 0x7f44640769f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f447c006fd0 tx=0x7f447c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:23.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.689+0000 7f447b7fe640 1 -- 172.21.15.28:0/599013660 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f448408e430 con 0x7f44840b9920 2024-04-25T19:12:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.885+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f44840d4820 con 0x7f44840b9920 2024-04-25T19:12:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.885+0000 7f447b7fe640 1 -- 172.21.15.28:0/599013660 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f446c05ed90 con 0x7f44840b9920 2024-04-25T19:12:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-04-25T19:12:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f44640744e0 msgr2=0x7f44640769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f44640744e0 0x7f44640769f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f447c006fd0 tx=0x7f447c008040 comp rx=0 tx=0).stop 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 msgr2=0x7f44840cd700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840cd700 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f446c00c470 tx=0x7f446c00c930 comp rx=0 tx=0).stop 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 shutdown_connections 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f44640744e0 0x7f44640769f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4484160310 0x7f44840cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f44840b9920 0x7f44840cd700 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 --2- 172.21.15.28:0/599013660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f44840b79f0 0x7f448414b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 >> 172.21.15.28:0/599013660 conn(0x7f44840d2a50 msgr2=0x7f44840b55f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 shutdown_connections 2024-04-25T19:12:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:23.887+0000 7f448960c640 1 -- 172.21.15.28:0/599013660 wait complete. 2024-04-25T19:12:23.902 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T19:12:23.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:23.902 DEBUG:teuthology.orchestra.run.smithi053:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-25T19:12:23.933 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:12:23.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:23.934 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T19:12:24.002 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:24.002 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:24.018 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:24.018 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:24.074 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:24.074 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:24.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.279+0000 7f915fdb7640 1 Processor -- start 2024-04-25T19:12:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.279+0000 7f915fdb7640 1 -- start start 2024-04-25T19:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915fdb7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9158147150 0x7f9158146bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915fdb7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f915806cf80 0x7f91580d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915fdb7640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f91580d0fc0 0x7f9158161a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915fdb7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f915806df00 con 0x7f9158147150 2024-04-25T19:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915fdb7640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f91580cea40 con 0x7f91580d0fc0 2024-04-25T19:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915fdb7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f915806d390 con 0x7f915806cf80 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915db53640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9158147150 0x7f9158146bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915db53640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9158147150 0x7f9158146bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38258/0 (socket says 172.21.15.28:38258) 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915db53640 1 -- 172.21.15.28:0/1856999080 learned_addr learned my addr 172.21.15.28:0/1856999080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915e354640 1 --2- 172.21.15.28:0/1856999080 >> v2:172.21.15.53:3300/0 conn(0x7f91580d0fc0 0x7f9158161a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.281+0000 7f915d352640 1 --2- 172.21.15.28:0/1856999080 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f915806cf80 0x7f91580d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.282+0000 7f915db53640 1 -- 172.21.15.28:0/1856999080 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f915806cf80 msgr2=0x7f91580d0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.282+0000 7f915db53640 1 --2- 172.21.15.28:0/1856999080 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f915806cf80 0x7f91580d0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.282+0000 7f915db53640 1 -- 172.21.15.28:0/1856999080 >> v2:172.21.15.53:3300/0 conn(0x7f91580d0fc0 msgr2=0x7f9158161a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.282+0000 7f915db53640 1 --2- 172.21.15.28:0/1856999080 >> v2:172.21.15.53:3300/0 conn(0x7f91580d0fc0 0x7f9158161a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.282+0000 7f915db53640 1 -- 172.21.15.28:0/1856999080 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915806d660 con 0x7f9158147150 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.282+0000 7f915db53640 1 --2- 172.21.15.28:0/1856999080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9158147150 0x7f9158146bf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9158072e40 tx=0x7f915002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d09a146326756f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.282+0000 7f915cb51640 1 -- 172.21.15.28:0/1856999080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150037070 con 0x7f9158147150 2024-04-25T19:12:24.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915cb51640 1 -- 172.21.15.28:0/1856999080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9150032440 con 0x7f9158147150 2024-04-25T19:12:24.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915cb51640 1 -- 172.21.15.28:0/1856999080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150032810 con 0x7f9158147150 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 -- 172.21.15.28:0/1856999080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9158147150 msgr2=0x7f9158146bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 --2- 172.21.15.28:0/1856999080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9158147150 0x7f9158146bf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f9158072e40 tx=0x7f915002a140 comp rx=0 tx=0).stop 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 -- 172.21.15.28:0/1856999080 shutdown_connections 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 --2- 172.21.15.28:0/1856999080 >> v2:172.21.15.53:3300/0 conn(0x7f91580d0fc0 0x7f9158161a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 --2- 172.21.15.28:0/1856999080 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f915806cf80 0x7f91580d0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 --2- 172.21.15.28:0/1856999080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9158147150 0x7f9158146bf0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 -- 172.21.15.28:0/1856999080 >> 172.21.15.28:0/1856999080 conn(0x7f9158073670 msgr2=0x7f9158073a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 -- 172.21.15.28:0/1856999080 shutdown_connections 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.283+0000 7f915fdb7640 1 -- 172.21.15.28:0/1856999080 wait complete. 2024-04-25T19:12:24.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.285+0000 7f915fdb7640 1 Processor -- start 2024-04-25T19:12:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.285+0000 7f915fdb7640 1 -- start start 2024-04-25T19:12:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.286+0000 7f915fdb7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806cf80 0x7f915815d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.286+0000 7f915fdb7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580d0fc0 0x7f91581e7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915fdb7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9158147150 0x7f91580ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915fdb7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91580d13d0 con 0x7f915806cf80 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915fdb7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f915806dc20 con 0x7f91580d0fc0 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915fdb7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9158161f40 con 0x7f9158147150 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806cf80 0x7f915815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806cf80 0x7f915815d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38260/0 (socket says 172.21.15.28:38260) 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 -- 172.21.15.28:0/3442903912 learned_addr learned my addr 172.21.15.28:0/3442903912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915d352640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580d0fc0 0x7f91581e7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915e354640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9158147150 0x7f91580ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 -- 172.21.15.28:0/3442903912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9158147150 msgr2=0x7f91580ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9158147150 0x7f91580ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 -- 172.21.15.28:0/3442903912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580d0fc0 msgr2=0x7f91581e7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580d0fc0 0x7f91581e7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 -- 172.21.15.28:0/3442903912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91581e8390 con 0x7f915806cf80 2024-04-25T19:12:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915db53640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806cf80 0x7f915815d400 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f915002a180 tx=0x7f91500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f91567fc640 1 -- 172.21.15.28:0/3442903912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150041020 con 0x7f915806cf80 2024-04-25T19:12:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f91567fc640 1 -- 172.21.15.28:0/3442903912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f915003d620 con 0x7f915806cf80 2024-04-25T19:12:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91581e8670 con 0x7f915806cf80 2024-04-25T19:12:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f91567fc640 1 -- 172.21.15.28:0/3442903912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9150037040 con 0x7f915806cf80 2024-04-25T19:12:24.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.287+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91581e8c30 con 0x7f915806cf80 2024-04-25T19:12:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.288+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9158050740 con 0x7f915806cf80 2024-04-25T19:12:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.289+0000 7f91567fc640 1 -- 172.21.15.28:0/3442903912 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9150044020 con 0x7f915806cf80 2024-04-25T19:12:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.289+0000 7f91567fc640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91440744b0 0x7f91440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.289+0000 7f91567fc640 1 -- 172.21.15.28:0/3442903912 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f91500b5e40 con 0x7f915806cf80 2024-04-25T19:12:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.290+0000 7f915d352640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91440744b0 0x7f91440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.290+0000 7f915d352640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91440744b0 0x7f91440769c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f91581427c0 tx=0x7f914c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.294+0000 7f91567fc640 1 -- 172.21.15.28:0/3442903912 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f915004b030 con 0x7f915806cf80 2024-04-25T19:12:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.481+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9158147830 con 0x7f915806cf80 2024-04-25T19:12:24.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.483+0000 7f91567fc640 1 -- 172.21.15.28:0/3442903912 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f9150080130 con 0x7f915806cf80 2024-04-25T19:12:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:24.485 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91440744b0 msgr2=0x7f91440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91440744b0 0x7f91440769c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f91581427c0 tx=0x7f914c008040 comp rx=0 tx=0).stop 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806cf80 msgr2=0x7f915815d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806cf80 0x7f915815d400 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f915002a180 tx=0x7f91500046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 shutdown_connections 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91440744b0 0x7f91440769c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9158147150 0x7f91580ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91580d0fc0 0x7f91581e7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 --2- 172.21.15.28:0/3442903912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915806cf80 0x7f915815d400 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 >> 172.21.15.28:0/3442903912 conn(0x7f9158073670 msgr2=0x7f91580cedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 shutdown_connections 2024-04-25T19:12:24.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.485+0000 7f915fdb7640 1 -- 172.21.15.28:0/3442903912 wait complete. 2024-04-25T19:12:24.498 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:24.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:24.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.611+0000 7f8b03d00640 1 Processor -- start 2024-04-25T19:12:24.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.611+0000 7f8b03d00640 1 -- start start 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.612+0000 7f8b03d00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc0d3250 0x7f8afc0cd630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.612+0000 7f8b03d00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8afc1604c0 0x7f8afc1608d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.612+0000 7f8b03d00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8afc0cdb70 con 0x7f8afc06ed00 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.612+0000 7f8b03d00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8afc0d2dc0 con 0x7f8afc1604c0 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.612+0000 7f8b03d00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8afc161990 con 0x7f8afc0d3250 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0229d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8afc1604c0 0x7f8afc1608d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0229d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8afc1604c0 0x7f8afc1608d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38274/0 (socket says 172.21.15.28:38274) 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0229d640 1 -- 172.21.15.28:0/3963438409 learned_addr learned my addr 172.21.15.28:0/3963438409 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0129b640 1 --2- 172.21.15.28:0/3963438409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc0d3250 0x7f8afc0cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0229d640 1 -- 172.21.15.28:0/3963438409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc0d3250 msgr2=0x7f8afc0cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0229d640 1 --2- 172.21.15.28:0/3963438409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc0d3250 0x7f8afc0cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0229d640 1 -- 172.21.15.28:0/3963438409 >> v1:172.21.15.53:6789/0 conn(0x7f8afc06ed00 legacy=0x7f8afc0ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0229d640 1 -- 172.21.15.28:0/3963438409 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc0d24b0 con 0x7f8afc1604c0 2024-04-25T19:12:24.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.613+0000 7f8b0129b640 1 --2- 172.21.15.28:0/3963438409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc0d3250 0x7f8afc0cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.614+0000 7f8b0229d640 1 --2- 172.21.15.28:0/3963438409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc1604c0 0x7f8afc1608d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009e50 tx=0x7f8af402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c63a4871215003 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.614+0000 7f8b00a9a640 1 -- 172.21.15.28:0/3963438409 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4036070 con 0x7f8afc1604c0 2024-04-25T19:12:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.614+0000 7f8b00a9a640 1 -- 172.21.15.28:0/3963438409 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af4031600 con 0x7f8afc1604c0 2024-04-25T19:12:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.614+0000 7f8b00a9a640 1 -- 172.21.15.28:0/3963438409 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af40319d0 con 0x7f8afc1604c0 2024-04-25T19:12:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.614+0000 7f8b03d00640 1 -- 172.21.15.28:0/3963438409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc1604c0 msgr2=0x7f8afc1608d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.614+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3963438409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc1604c0 0x7f8afc1608d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f8af4009e50 tx=0x7f8af402a710 comp rx=0 tx=0).stop 2024-04-25T19:12:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.615+0000 7f8b03d00640 1 -- 172.21.15.28:0/3963438409 shutdown_connections 2024-04-25T19:12:24.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.615+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3963438409 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc1604c0 0x7f8afc1608d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.615+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3963438409 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc0d3250 0x7f8afc0cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.615+0000 7f8b03d00640 1 -- 172.21.15.28:0/3963438409 >> 172.21.15.28:0/3963438409 conn(0x7f8afc0b7640 msgr2=0x7f8afc0b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.615+0000 7f8b03d00640 1 -- 172.21.15.28:0/3963438409 shutdown_connections 2024-04-25T19:12:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.615+0000 7f8b03d00640 1 -- 172.21.15.28:0/3963438409 wait complete. 2024-04-25T19:12:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.616+0000 7f8b03d00640 1 Processor -- start 2024-04-25T19:12:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.616+0000 7f8b03d00640 1 -- start start 2024-04-25T19:12:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b03d00640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc06ed00 0x7f8afc09f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b03d00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc0d3250 0x7f8afc1580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b03d00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8afc1604c0 0x7f8afc1585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b03d00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc0d3660 con 0x7f8afc0d3250 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b03d00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8afc0d2a50 con 0x7f8afc1604c0 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b03d00640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8afc0cf110 con 0x7f8afc06ed00 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc0d3250 0x7f8afc1580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc0d3250 0x7f8afc1580a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38280/0 (socket says 172.21.15.28:38280) 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 -- 172.21.15.28:0/3792024297 learned_addr learned my addr 172.21.15.28:0/3792024297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b01a9c640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc06ed00 0x7f8afc09f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0229d640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8afc1604c0 0x7f8afc1585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 -- 172.21.15.28:0/3792024297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc06ed00 msgr2=0x7f8afc09f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc06ed00 0x7f8afc09f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 -- 172.21.15.28:0/3792024297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8afc1604c0 msgr2=0x7f8afc1585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8afc1604c0 0x7f8afc1585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 -- 172.21.15.28:0/3792024297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8afc159460 con 0x7f8afc0d3250 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.618+0000 7f8b0129b640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc0d3250 0x7f8afc1580a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00d800 tx=0x7f8aec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.619+0000 7f8af27fc640 1 -- 172.21.15.28:0/3792024297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec004030 con 0x7f8afc0d3250 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.619+0000 7f8af27fc640 1 -- 172.21.15.28:0/3792024297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aec0041d0 con 0x7f8afc0d3250 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.619+0000 7f8af27fc640 1 -- 172.21.15.28:0/3792024297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0096a0 con 0x7f8afc0d3250 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.619+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8afc15b900 con 0x7f8afc0d3250 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.619+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8afc15bec0 con 0x7f8afc0d3250 2024-04-25T19:12:24.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.620+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8afc050740 con 0x7f8afc0d3250 2024-04-25T19:12:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.620+0000 7f8af27fc640 1 -- 172.21.15.28:0/3792024297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8aec009840 con 0x7f8afc0d3250 2024-04-25T19:12:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.621+0000 7f8af27fc640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ae00744e0 0x7f8ae00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.621+0000 7f8af27fc640 1 -- 172.21.15.28:0/3792024297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8aec095710 con 0x7f8afc0d3250 2024-04-25T19:12:24.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.621+0000 7f8b01a9c640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ae00744e0 0x7f8ae00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.621+0000 7f8b01a9c640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ae00744e0 0x7f8ae00769f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8af80016e0 tx=0x7f8af8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.626+0000 7f8af27fc640 1 -- 172.21.15.28:0/3792024297 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8aec05f4e0 con 0x7f8afc0d3250 2024-04-25T19:12:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.808+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8afc15b690 con 0x7f8afc0d3250 2024-04-25T19:12:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.809+0000 7f8af27fc640 1 -- 172.21.15.28:0/3792024297 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f8aec0637e0 con 0x7f8afc0d3250 2024-04-25T19:12:24.811 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ae00744e0 msgr2=0x7f8ae00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ae00744e0 0x7f8ae00769f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8af80016e0 tx=0x7f8af8009040 comp rx=0 tx=0).stop 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc0d3250 msgr2=0x7f8afc1580a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc0d3250 0x7f8afc1580a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f8aec00d800 tx=0x7f8aec00dcc0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 shutdown_connections 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8ae00744e0 0x7f8ae00769f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8afc1604c0 0x7f8afc1585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8afc0d3250 0x7f8afc1580a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 --2- 172.21.15.28:0/3792024297 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8afc06ed00 0x7f8afc09f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 >> 172.21.15.28:0/3792024297 conn(0x7f8afc0b7640 msgr2=0x7f8afc070650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 shutdown_connections 2024-04-25T19:12:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.811+0000 7f8b03d00640 1 -- 172.21.15.28:0/3792024297 wait complete. 2024-04-25T19:12:24.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.938+0000 7f06933b7640 1 Processor -- start 2024-04-25T19:12:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.939+0000 7f06933b7640 1 -- start start 2024-04-25T19:12:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f06933b7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f068c0d2c60 0x7f068c0d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f06933b7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f068c164090 con 0x7f068c160180 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f06933b7640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f068c164bb0 con 0x7f068c163c80 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f06933b7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f068c0d0c00 con 0x7f068c0d2c60 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f0691954640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f068c0d2c60 0x7f068c0d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f0691954640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f068c0d2c60 0x7f068c0d06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38294/0 (socket says 172.21.15.28:38294) 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f0691153640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f068c160180 0x7f068c0b9090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36878/0 (socket says 172.21.15.28:36878) 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f0691954640 1 -- 172.21.15.28:0/2267787274 learned_addr learned my addr 172.21.15.28:0/2267787274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.940+0000 7f0691954640 1 -- 172.21.15.28:0/2267787274 >> v1:172.21.15.53:6789/0 conn(0x7f068c163c80 legacy=0x7f068c0d2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.941+0000 7f0691954640 1 -- 172.21.15.28:0/2267787274 >> v1:172.21.15.28:6789/0 conn(0x7f068c160180 legacy=0x7f068c0b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.941+0000 7f0691954640 1 -- 172.21.15.28:0/2267787274 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c164310 con 0x7f068c0d2c60 2024-04-25T19:12:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.941+0000 7f0691954640 1 --2- 172.21.15.28:0/2267787274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0d2c60 0x7f068c0d06c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0684009e50 tx=0x7f068402a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ed0399e95c3847 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.941+0000 7f0683fff640 1 -- 172.21.15.28:0/2267787274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0684036070 con 0x7f068c0d2c60 2024-04-25T19:12:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.941+0000 7f0683fff640 1 -- 172.21.15.28:0/2267787274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0684031600 con 0x7f068c0d2c60 2024-04-25T19:12:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.941+0000 7f0683fff640 1 -- 172.21.15.28:0/2267787274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06840319f0 con 0x7f068c0d2c60 2024-04-25T19:12:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.942+0000 7f06933b7640 1 -- 172.21.15.28:0/2267787274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0d2c60 msgr2=0x7f068c0d06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.942+0000 7f06933b7640 1 --2- 172.21.15.28:0/2267787274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0d2c60 0x7f068c0d06c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0684009e50 tx=0x7f068402a7c0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.942+0000 7f06933b7640 1 -- 172.21.15.28:0/2267787274 shutdown_connections 2024-04-25T19:12:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.942+0000 7f06933b7640 1 --2- 172.21.15.28:0/2267787274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0d2c60 0x7f068c0d06c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.942+0000 7f06933b7640 1 -- 172.21.15.28:0/2267787274 >> 172.21.15.28:0/2267787274 conn(0x7f068c06c750 msgr2=0x7f068c06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.942+0000 7f06933b7640 1 -- 172.21.15.28:0/2267787274 shutdown_connections 2024-04-25T19:12:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.942+0000 7f06933b7640 1 -- 172.21.15.28:0/2267787274 wait complete. 2024-04-25T19:12:24.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.944+0000 7f06933b7640 1 Processor -- start 2024-04-25T19:12:24.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.944+0000 7f06933b7640 1 -- start start 2024-04-25T19:12:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f06933b7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f068c0d2c60 0x7f068c14cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f06933b7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c160180 0x7f068c0c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f06933b7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f068c163c80 0x7f068c1621d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f06933b7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1648d0 con 0x7f068c160180 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f06933b7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f068c0d4510 con 0x7f068c0d2c60 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f06933b7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f068c0d49a0 con 0x7f068c163c80 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f0690952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c160180 0x7f068c0c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f0690952640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c160180 0x7f068c0c8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38306/0 (socket says 172.21.15.28:38306) 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.945+0000 7f0690952640 1 -- 172.21.15.28:0/3331453337 learned_addr learned my addr 172.21.15.28:0/3331453337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0691153640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f068c0d2c60 0x7f068c14cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0691954640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f068c163c80 0x7f068c1621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0690952640 1 -- 172.21.15.28:0/3331453337 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f068c163c80 msgr2=0x7f068c1621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0690952640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f068c163c80 0x7f068c1621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0690952640 1 -- 172.21.15.28:0/3331453337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f068c0d2c60 msgr2=0x7f068c14cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0690952640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f068c0d2c60 0x7f068c14cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0690952640 1 -- 172.21.15.28:0/3331453337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c0ceea0 con 0x7f068c160180 2024-04-25T19:12:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0690952640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c160180 0x7f068c0c8690 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f067c00c3f0 tx=0x7f067c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0681ffb640 1 -- 172.21.15.28:0/3331453337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c0052d0 con 0x7f068c160180 2024-04-25T19:12:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0681ffb640 1 -- 172.21.15.28:0/3331453337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f067c005490 con 0x7f068c160180 2024-04-25T19:12:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c14d770 con 0x7f068c160180 2024-04-25T19:12:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f0681ffb640 1 -- 172.21.15.28:0/3331453337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c007260 con 0x7f068c160180 2024-04-25T19:12:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.946+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c0cf160 con 0x7f068c160180 2024-04-25T19:12:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.947+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068c050740 con 0x7f068c160180 2024-04-25T19:12:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.948+0000 7f0681ffb640 1 -- 172.21.15.28:0/3331453337 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f067c007400 con 0x7f068c160180 2024-04-25T19:12:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.948+0000 7f0681ffb640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f06700744b0 0x7f06700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.949+0000 7f0681ffb640 1 -- 172.21.15.28:0/3331453337 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f067c094ae0 con 0x7f068c160180 2024-04-25T19:12:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.949+0000 7f0691153640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f06700744b0 0x7f06700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.949+0000 7f0691153640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f06700744b0 0x7f06700769c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0688000f60 tx=0x7f0688000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:24.953+0000 7f0681ffb640 1 -- 172.21.15.28:0/3331453337 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f067c05e780 con 0x7f068c160180 2024-04-25T19:12:25.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.138+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f068c0d3f90 con 0x7f068c160180 2024-04-25T19:12:25.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.139+0000 7f0681ffb640 1 -- 172.21.15.28:0/3331453337 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f067c062a80 con 0x7f068c160180 2024-04-25T19:12:25.141 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f06700744b0 msgr2=0x7f06700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f06700744b0 0x7f06700769c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0688000f60 tx=0x7f0688000ed0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c160180 msgr2=0x7f068c0c8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c160180 0x7f068c0c8690 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f067c00c3f0 tx=0x7f067c00c8b0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 shutdown_connections 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f06700744b0 0x7f06700769c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f068c163c80 0x7f068c1621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c160180 0x7f068c0c8690 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 --2- 172.21.15.28:0/3331453337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f068c0d2c60 0x7f068c14cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 >> 172.21.15.28:0/3331453337 conn(0x7f068c06c750 msgr2=0x7f068c0b9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 shutdown_connections 2024-04-25T19:12:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.141+0000 7f06933b7640 1 -- 172.21.15.28:0/3331453337 wait complete. 2024-04-25T19:12:25.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.268+0000 7f83f6784640 1 Processor -- start 2024-04-25T19:12:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.268+0000 7f83f6784640 1 -- start start 2024-04-25T19:12:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f6784640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83f0167850 0x7f83f0169c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f6784640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83f00d1ce0 con 0x7f83f009cf00 2024-04-25T19:12:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f6784640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83f00cf8d0 con 0x7f83f009caf0 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f6784640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f83f00b8fa0 con 0x7f83f0167850 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f4d21640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83f0167850 0x7f83f0169c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f4d21640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83f0167850 0x7f83f0169c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38314/0 (socket says 172.21.15.28:38314) 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83ef7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f83f009cf00 0x7f83f00b71d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36890/0 (socket says 172.21.15.28:36890) 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f4d21640 1 -- 172.21.15.28:0/1990373358 learned_addr learned my addr 172.21.15.28:0/1990373358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f4d21640 1 -- 172.21.15.28:0/1990373358 >> v1:172.21.15.53:6789/0 conn(0x7f83f009caf0 legacy=0x7f83f00d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.269+0000 7f83f4d21640 1 -- 172.21.15.28:0/1990373358 >> v1:172.21.15.28:6789/0 conn(0x7f83f009cf00 legacy=0x7f83f00b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.270+0000 7f83eeffd640 1 -- 172.21.15.28:0/1990373358 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2136034776 0 0) 0x7f83f00d1ce0 con 0x7f83f009cf00 2024-04-25T19:12:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.270+0000 7f83f4d21640 1 -- 172.21.15.28:0/1990373358 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f00cfbb0 con 0x7f83f0167850 2024-04-25T19:12:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.270+0000 7f83f4d21640 1 --2- 172.21.15.28:0/1990373358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0167850 0x7f83f0169c70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009e50 tx=0x7f83e802a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7800ccef195606f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.270+0000 7f83eeffd640 1 -- 172.21.15.28:0/1990373358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e8036070 con 0x7f83f0167850 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.270+0000 7f83eeffd640 1 -- 172.21.15.28:0/1990373358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e8031600 con 0x7f83f0167850 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.270+0000 7f83eeffd640 1 -- 172.21.15.28:0/1990373358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e80319d0 con 0x7f83f0167850 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.271+0000 7f83f6784640 1 -- 172.21.15.28:0/1990373358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0167850 msgr2=0x7f83f0169c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.271+0000 7f83f6784640 1 --2- 172.21.15.28:0/1990373358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0167850 0x7f83f0169c70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009e50 tx=0x7f83e802a6d0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.271+0000 7f83f6784640 1 -- 172.21.15.28:0/1990373358 shutdown_connections 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.271+0000 7f83f6784640 1 --2- 172.21.15.28:0/1990373358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f0167850 0x7f83f0169c70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.271+0000 7f83f6784640 1 -- 172.21.15.28:0/1990373358 >> 172.21.15.28:0/1990373358 conn(0x7f83f00d03f0 msgr2=0x7f83f00d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.271+0000 7f83f6784640 1 -- 172.21.15.28:0/1990373358 shutdown_connections 2024-04-25T19:12:25.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.271+0000 7f83f6784640 1 -- 172.21.15.28:0/1990373358 wait complete. 2024-04-25T19:12:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.273+0000 7f83f6784640 1 Processor -- start 2024-04-25T19:12:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.273+0000 7f83f6784640 1 -- start start 2024-04-25T19:12:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.274+0000 7f83f6784640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f009caf0 0x7f83f015add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.274+0000 7f83f6784640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f009cf00 0x7f83f00b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.274+0000 7f83f6784640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83f0167850 0x7f83f00b8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.274+0000 7f83f6784640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83f00b78e0 con 0x7f83f009cf00 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.274+0000 7f83f6784640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f83f00d19e0 con 0x7f83f009caf0 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.274+0000 7f83f6784640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f83f00b9430 con 0x7f83f0167850 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.274+0000 7f83ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f009cf00 0x7f83f00b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.274+0000 7f83ef7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f009cf00 0x7f83f00b7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38324/0 (socket says 172.21.15.28:38324) 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ef7fe640 1 -- 172.21.15.28:0/4083872141 learned_addr learned my addr 172.21.15.28:0/4083872141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83effff640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f009caf0 0x7f83f015add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83f4d21640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83f0167850 0x7f83f00b8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ef7fe640 1 -- 172.21.15.28:0/4083872141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83f0167850 msgr2=0x7f83f00b8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ef7fe640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83f0167850 0x7f83f00b8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ef7fe640 1 -- 172.21.15.28:0/4083872141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f009caf0 msgr2=0x7f83f015add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ef7fe640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f009caf0 0x7f83f015add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ef7fe640 1 -- 172.21.15.28:0/4083872141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83f00c7c00 con 0x7f83f009cf00 2024-04-25T19:12:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ef7fe640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f009cf00 0x7f83f00b7af0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f83e400c3f0 tx=0x7f83e400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ecff9640 1 -- 172.21.15.28:0/4083872141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e40074e0 con 0x7f83f009cf00 2024-04-25T19:12:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ecff9640 1 -- 172.21.15.28:0/4083872141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e40047e0 con 0x7f83f009cf00 2024-04-25T19:12:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83f0165690 con 0x7f83f009cf00 2024-04-25T19:12:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83ecff9640 1 -- 172.21.15.28:0/4083872141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4004bd0 con 0x7f83f009cf00 2024-04-25T19:12:25.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.275+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83f00c81c0 con 0x7f83f009cf00 2024-04-25T19:12:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.276+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83f0050740 con 0x7f83f009cf00 2024-04-25T19:12:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.277+0000 7f83ecff9640 1 -- 172.21.15.28:0/4083872141 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f83e401a020 con 0x7f83f009cf00 2024-04-25T19:12:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.277+0000 7f83ecff9640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83dc0744b0 0x7f83dc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.278+0000 7f83effff640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83dc0744b0 0x7f83dc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.278+0000 7f83ecff9640 1 -- 172.21.15.28:0/4083872141 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f83e4094cb0 con 0x7f83f009cf00 2024-04-25T19:12:25.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.278+0000 7f83effff640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83dc0744b0 0x7f83dc0769c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f83f0069f40 tx=0x7f83e0009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.282+0000 7f83ecff9640 1 -- 172.21.15.28:0/4083872141 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f83e405ea80 con 0x7f83f009cf00 2024-04-25T19:12:25.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.466+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f83f00d2410 con 0x7f83f009cf00 2024-04-25T19:12:25.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.467+0000 7f83ecff9640 1 -- 172.21.15.28:0/4083872141 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f83e4062d80 con 0x7f83f009cf00 2024-04-25T19:12:25.469 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.468+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83dc0744b0 msgr2=0x7f83dc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.468+0000 7f83f6784640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83dc0744b0 0x7f83dc0769c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f83f0069f40 tx=0x7f83e0009060 comp rx=0 tx=0).stop 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.468+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f009cf00 msgr2=0x7f83f00b7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.468+0000 7f83f6784640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f009cf00 0x7f83f00b7af0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f83e400c3f0 tx=0x7f83e400c8b0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.469+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 shutdown_connections 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.469+0000 7f83f6784640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83dc0744b0 0x7f83dc0769c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.469+0000 7f83f6784640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83f0167850 0x7f83f00b8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.469+0000 7f83f6784640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83f009cf00 0x7f83f00b7af0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.469+0000 7f83f6784640 1 --2- 172.21.15.28:0/4083872141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f009caf0 0x7f83f015add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.469+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 >> 172.21.15.28:0/4083872141 conn(0x7f83f00d03f0 msgr2=0x7f83f0072c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.469+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 shutdown_connections 2024-04-25T19:12:25.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.469+0000 7f83f6784640 1 -- 172.21.15.28:0/4083872141 wait complete. 2024-04-25T19:12:25.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.595+0000 7f24eeef9640 1 Processor -- start 2024-04-25T19:12:25.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.595+0000 7f24eeef9640 1 -- start start 2024-04-25T19:12:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.596+0000 7f24eeef9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b4c80 0x7f24e8160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.596+0000 7f24eeef9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24e8164800 0x7f24e80b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.596+0000 7f24eeef9640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24e80d15b0 con 0x7f24e80b78c0 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.596+0000 7f24eeef9640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f24e8164c10 con 0x7f24e8164800 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.596+0000 7f24eeef9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f24e80d12d0 con 0x7f24e80b4c80 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.596+0000 7f24e7fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24e8164800 0x7f24e80b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.596+0000 7f24ed496640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f24e80b78c0 0x7f24e80b7cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36904/0 (socket says 172.21.15.28:36904) 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.596+0000 7f24ed496640 1 -- 172.21.15.28:0/364721206 learned_addr learned my addr 172.21.15.28:0/364721206 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.597+0000 7f24ecc95640 1 --2- 172.21.15.28:0/364721206 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b4c80 0x7f24e8160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.597+0000 7f24e7fff640 1 -- 172.21.15.28:0/364721206 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b4c80 msgr2=0x7f24e8160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.597+0000 7f24e7fff640 1 --2- 172.21.15.28:0/364721206 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b4c80 0x7f24e8160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.597+0000 7f24e7fff640 1 -- 172.21.15.28:0/364721206 >> v1:172.21.15.28:6789/0 conn(0x7f24e80b78c0 legacy=0x7f24e80b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.597+0000 7f24e77fe640 1 -- 172.21.15.28:0/364721206 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 747065878 0 0) 0x7f24e80d15b0 con 0x7f24e80b78c0 2024-04-25T19:12:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.597+0000 7f24e7fff640 1 -- 172.21.15.28:0/364721206 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e8163d20 con 0x7f24e8164800 2024-04-25T19:12:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24e7fff640 1 --2- 172.21.15.28:0/364721206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e8164800 0x7f24e80b46c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009790 tx=0x7f24d402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8783090739818791 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24e77fe640 1 -- 172.21.15.28:0/364721206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d4037070 con 0x7f24e8164800 2024-04-25T19:12:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24e77fe640 1 -- 172.21.15.28:0/364721206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24d4032440 con 0x7f24e8164800 2024-04-25T19:12:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24e77fe640 1 -- 172.21.15.28:0/364721206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24d4032810 con 0x7f24e8164800 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24eeef9640 1 -- 172.21.15.28:0/364721206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e8164800 msgr2=0x7f24e80b46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24eeef9640 1 --2- 172.21.15.28:0/364721206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e8164800 0x7f24e80b46c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f24d4009790 tx=0x7f24d402a180 comp rx=0 tx=0).stop 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24eeef9640 1 -- 172.21.15.28:0/364721206 shutdown_connections 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24eeef9640 1 --2- 172.21.15.28:0/364721206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e8164800 0x7f24e80b46c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24eeef9640 1 --2- 172.21.15.28:0/364721206 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b4c80 0x7f24e8160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.598+0000 7f24eeef9640 1 -- 172.21.15.28:0/364721206 >> 172.21.15.28:0/364721206 conn(0x7f24e80d2980 msgr2=0x7f24e80d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.599+0000 7f24eeef9640 1 -- 172.21.15.28:0/364721206 shutdown_connections 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.599+0000 7f24eeef9640 1 -- 172.21.15.28:0/364721206 wait complete. 2024-04-25T19:12:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.600+0000 7f24eeef9640 1 Processor -- start 2024-04-25T19:12:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.600+0000 7f24eeef9640 1 -- start start 2024-04-25T19:12:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24eeef9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e80b4c80 0x7f24e80c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24eeef9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b78c0 0x7f24e815c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24eeef9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24e8164800 0x7f24e815dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24eeef9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f24e80b5240 con 0x7f24e80b4c80 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24eeef9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f24e81642e0 con 0x7f24e8164800 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24eeef9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f24e80b53c0 con 0x7f24e80b78c0 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e80b4c80 0x7f24e80c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e80b4c80 0x7f24e80c8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38356/0 (socket says 172.21.15.28:38356) 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 -- 172.21.15.28:0/3420865167 learned_addr learned my addr 172.21.15.28:0/3420865167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24e7fff640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b78c0 0x7f24e815c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ed496640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24e8164800 0x7f24e815dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 -- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b78c0 msgr2=0x7f24e815c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b78c0 0x7f24e815c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 -- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24e8164800 msgr2=0x7f24e815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24e8164800 0x7f24e815dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 -- 172.21.15.28:0/3420865167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24e8160a60 con 0x7f24e80b4c80 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24e7fff640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b78c0 0x7f24e815c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ed496640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24e8164800 0x7f24e815dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.602+0000 7f24ecc95640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e80b4c80 0x7f24e80c8450 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00e870 tx=0x7f24dc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.603+0000 7f24e57fa640 1 -- 172.21.15.28:0/3420865167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc017070 con 0x7f24e80b4c80 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.603+0000 7f24e57fa640 1 -- 172.21.15.28:0/3420865167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24dc00c940 con 0x7f24e80b4c80 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.603+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24e8161b00 con 0x7f24e80b4c80 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.603+0000 7f24e57fa640 1 -- 172.21.15.28:0/3420865167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24dc00cd30 con 0x7f24e80b4c80 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.603+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24e8160d40 con 0x7f24e80b4c80 2024-04-25T19:12:25.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.604+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24e8050740 con 0x7f24e80b4c80 2024-04-25T19:12:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.604+0000 7f24e57fa640 1 -- 172.21.15.28:0/3420865167 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f24dc01a070 con 0x7f24e80b4c80 2024-04-25T19:12:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.605+0000 7f24e57fa640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24cc0744b0 0x7f24cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.605+0000 7f24e57fa640 1 -- 172.21.15.28:0/3420865167 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f24dc095470 con 0x7f24e80b4c80 2024-04-25T19:12:25.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.605+0000 7f24e7fff640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24cc0744b0 0x7f24cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.605+0000 7f24e7fff640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24cc0744b0 0x7f24cc0769c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f24d4034000 tx=0x7f24d402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.609+0000 7f24e57fa640 1 -- 172.21.15.28:0/3420865167 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f24dc05f890 con 0x7f24e80b4c80 2024-04-25T19:12:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.793+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f24e80bafb0 con 0x7f24e80b4c80 2024-04-25T19:12:25.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.794+0000 7f24e57fa640 1 -- 172.21.15.28:0/3420865167 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f24dc05f190 con 0x7f24e80b4c80 2024-04-25T19:12:25.796 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24cc0744b0 msgr2=0x7f24cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24cc0744b0 0x7f24cc0769c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f24d4034000 tx=0x7f24d402e000 comp rx=0 tx=0).stop 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e80b4c80 msgr2=0x7f24e80c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e80b4c80 0x7f24e80c8450 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f24dc00e870 tx=0x7f24dc00ed30 comp rx=0 tx=0).stop 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 shutdown_connections 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f24cc0744b0 0x7f24cc0769c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24e8164800 0x7f24e815dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f24e80b78c0 0x7f24e815c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 --2- 172.21.15.28:0/3420865167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24e80b4c80 0x7f24e80c8450 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 >> 172.21.15.28:0/3420865167 conn(0x7f24e80d2980 msgr2=0x7f24e80d43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:25.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 shutdown_connections 2024-04-25T19:12:25.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.796+0000 7f24eeef9640 1 -- 172.21.15.28:0/3420865167 wait complete. 2024-04-25T19:12:25.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:12:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.921+0000 7f92042f2640 1 Processor -- start 2024-04-25T19:12:25.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.921+0000 7f92042f2640 1 -- start start 2024-04-25T19:12:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.922+0000 7f92042f2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc1616a0 0x7f91fc09b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.922+0000 7f92042f2640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f91fc0d1b30 0x7f91fc0d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.922+0000 7f92042f2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91fc160c00 con 0x7f91fc06edb0 2024-04-25T19:12:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.922+0000 7f92042f2640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f91fc09c190 con 0x7f91fc0d1b30 2024-04-25T19:12:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.922+0000 7f92042f2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f91fc09beb0 con 0x7f91fc1616a0 2024-04-25T19:12:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.922+0000 7f920208e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f91fc06edb0 0x7f91fc0b3c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36916/0 (socket says 172.21.15.28:36916) 2024-04-25T19:12:25.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.922+0000 7f920208e640 1 -- 172.21.15.28:0/1726011729 learned_addr learned my addr 172.21.15.28:0/1726011729 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:25.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920188d640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc1616a0 0x7f91fc09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920288f640 1 --2- 172.21.15.28:0/1726011729 >> v2:172.21.15.53:3300/0 conn(0x7f91fc0d1b30 0x7f91fc0d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1388328748 0 0) 0x7f91fc160c00 con 0x7f91fc06edb0 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91e8003520 con 0x7f91fc06edb0 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1142386942 0 0) 0x7f91e8003520 con 0x7f91fc06edb0 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc1616a0 msgr2=0x7f91fc09b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc1616a0 0x7f91fc09b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 >> v2:172.21.15.53:3300/0 conn(0x7f91fc0d1b30 msgr2=0x7f91fc0d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 --2- 172.21.15.28:0/1726011729 >> v2:172.21.15.53:3300/0 conn(0x7f91fc0d1b30 0x7f91fc0d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91fc160ea0 con 0x7f91fc06edb0 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f91f4001670 con 0x7f91fc06edb0 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 >> v1:172.21.15.28:6789/0 conn(0x7f91fc06edb0 legacy=0x7f91fc0b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc1616a0 0x7f91e80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e8003f30 0x7f91e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8004d40 0x7f91e8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91fc09c190 con 0x7f91e8003f30 2024-04-25T19:12:25.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f91fc09beb0 con 0x7f91e8004d40 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f91e800ba10 con 0x7f91fc1616a0 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920188d640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e8003f30 0x7f91e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920288f640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8004d40 0x7f91e8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.923+0000 7f920208e640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc1616a0 0x7f91e80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920188d640 1 -- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc1616a0 msgr2=0x7f91e80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920188d640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc1616a0 0x7f91e80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920188d640 1 -- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8004d40 msgr2=0x7f91e8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920188d640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8004d40 0x7f91e8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920188d640 1 -- 172.21.15.28:0/1726011729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e8003520 con 0x7f91e8003f30 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920288f640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8004d40 0x7f91e8005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920188d640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e8003f30 0x7f91e8004800 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f91fc163f40 tx=0x7f91e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934f0e6c893e1382 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec036070 con 0x7f91e8003f30 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ec031400 con 0x7f91e8003f30 2024-04-25T19:12:25.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.924+0000 7f920108c640 1 -- 172.21.15.28:0/1726011729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec009690 con 0x7f91e8003f30 2024-04-25T19:12:25.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.925+0000 7f92042f2640 1 -- 172.21.15.28:0/1726011729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e8003f30 msgr2=0x7f91e8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.925+0000 7f92042f2640 1 --2- 172.21.15.28:0/1726011729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91e8003f30 0x7f91e8004800 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f91fc163f40 tx=0x7f91e8005830 comp rx=0 tx=0).stop 2024-04-25T19:12:25.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.925+0000 7f920288f640 1 -- 172.21.15.28:0/1726011729 reap_dead start 2024-04-25T19:12:25.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.925+0000 7f92042f2640 1 -- 172.21.15.28:0/1726011729 shutdown_connections 2024-04-25T19:12:25.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.925+0000 7f92042f2640 1 -- 172.21.15.28:0/1726011729 >> 172.21.15.28:0/1726011729 conn(0x7f91fc0b7620 msgr2=0x7f91fc0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:25.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.925+0000 7f92042f2640 1 -- 172.21.15.28:0/1726011729 shutdown_connections 2024-04-25T19:12:25.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.925+0000 7f92042f2640 1 -- 172.21.15.28:0/1726011729 wait complete. 2024-04-25T19:12:25.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.927+0000 7f92042f2640 1 Processor -- start 2024-04-25T19:12:25.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.927+0000 7f92042f2640 1 -- start start 2024-04-25T19:12:25.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f92042f2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8003f30 0x7f91fc147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f92042f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91fc14a370 0x7f91fc0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f92042f2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc0b4880 0x7f91fc0af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f92042f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91fc09b540 con 0x7f91fc14a370 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f92042f2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f91fc0d2480 con 0x7f91e8003f30 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f92042f2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f91fc09b6c0 con 0x7f91fc0b4880 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f920188d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91fc14a370 0x7f91fc0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f920188d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91fc14a370 0x7f91fc0b4340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38364/0 (socket says 172.21.15.28:38364) 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.928+0000 7f920188d640 1 -- 172.21.15.28:0/198085472 learned_addr learned my addr 172.21.15.28:0/198085472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920208e640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8003f30 0x7f91fc147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920288f640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc0b4880 0x7f91fc0af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920188d640 1 -- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc0b4880 msgr2=0x7f91fc0af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920188d640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc0b4880 0x7f91fc0af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920188d640 1 -- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8003f30 msgr2=0x7f91fc147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:25.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920188d640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8003f30 0x7f91fc147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920188d640 1 -- 172.21.15.28:0/198085472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91fc0b4c90 con 0x7f91fc14a370 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920288f640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc0b4880 0x7f91fc0af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f920188d640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91fc14a370 0x7f91fc0b4340 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f91ec034040 tx=0x7f91ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f91f2ffd640 1 -- 172.21.15.28:0/198085472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec040020 con 0x7f91fc14a370 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f91f2ffd640 1 -- 172.21.15.28:0/198085472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91ec003a40 con 0x7f91fc14a370 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f91f2ffd640 1 -- 172.21.15.28:0/198085472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91ec036040 con 0x7f91fc14a370 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91fc14ae40 con 0x7f91fc14a370 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.929+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91fc148580 con 0x7f91fc14a370 2024-04-25T19:12:25.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.930+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91fc050740 con 0x7f91fc14a370 2024-04-25T19:12:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.931+0000 7f91f2ffd640 1 -- 172.21.15.28:0/198085472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f91ec043020 con 0x7f91fc14a370 2024-04-25T19:12:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.932+0000 7f91f2ffd640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91e00744b0 0x7f91e00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.932+0000 7f920208e640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91e00744b0 0x7f91e00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.932+0000 7f91f2ffd640 1 -- 172.21.15.28:0/198085472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f91ec0b5740 con 0x7f91fc14a370 2024-04-25T19:12:25.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.932+0000 7f920208e640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91e00744b0 0x7f91e00769c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f91f4001050 tx=0x7f91f4007970 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:25.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:25.937+0000 7f91f2ffd640 1 -- 172.21.15.28:0/198085472 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91ec04a070 con 0x7f91fc14a370 2024-04-25T19:12:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.140+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f91fc14abd0 con 0x7f91fc14a370 2024-04-25T19:12:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.141+0000 7f91f2ffd640 1 -- 172.21.15.28:0/198085472 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f91ec07fad0 con 0x7f91fc14a370 2024-04-25T19:12:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-04-25T19:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91e00744b0 msgr2=0x7f91e00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91e00744b0 0x7f91e00769c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f91f4001050 tx=0x7f91f4007970 comp rx=0 tx=0).stop 2024-04-25T19:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91fc14a370 msgr2=0x7f91fc0b4340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91fc14a370 0x7f91fc0b4340 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f91ec034040 tx=0x7f91ec004730 comp rx=0 tx=0).stop 2024-04-25T19:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 shutdown_connections 2024-04-25T19:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91e00744b0 0x7f91e00769c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91fc0b4880 0x7f91fc0af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91fc14a370 0x7f91fc0b4340 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 --2- 172.21.15.28:0/198085472 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91e8003f30 0x7f91fc147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 >> 172.21.15.28:0/198085472 conn(0x7f91fc0b7620 msgr2=0x7f91fc0b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 shutdown_connections 2024-04-25T19:12:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.143+0000 7f92042f2640 1 -- 172.21.15.28:0/198085472 wait complete. 2024-04-25T19:12:26.156 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-25T19:12:26.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:26.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-25T19:12:26.182 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:12:26.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:26.183 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T19:12:26.238 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:12:26.238 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:26.257 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:12:26.257 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:26.316 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:12:26.316 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:26.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.486+0000 7f6ece7ee640 1 Processor -- start 2024-04-25T19:12:26.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.486+0000 7f6ece7ee640 1 -- start start 2024-04-25T19:12:26.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6ece7ee640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6ec8163850 0x7f6ec809bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6ece7ee640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec80d1d30 con 0x7f6ec809cb20 2024-04-25T19:12:26.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6ece7ee640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec806ebb0 con 0x7f6ec80b91a0 2024-04-25T19:12:26.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6ece7ee640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f6ec80d2620 con 0x7f6ec8163850 2024-04-25T19:12:26.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6ec7fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6ec809cb20 0x7f6ec80ce010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36926/0 (socket says 172.21.15.28:36926) 2024-04-25T19:12:26.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6ec7fff640 1 -- 172.21.15.28:0/334283688 learned_addr learned my addr 172.21.15.28:0/334283688 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:26.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6eccd8b640 1 --2- 172.21.15.28:0/334283688 >> v2:172.21.15.28:3300/0 conn(0x7f6ec8163850 0x7f6ec809bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6eccd8b640 1 -- 172.21.15.28:0/334283688 >> v1:172.21.15.53:6789/0 conn(0x7f6ec80b91a0 legacy=0x7f6ec80d2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6eccd8b640 1 -- 172.21.15.28:0/334283688 >> v1:172.21.15.28:6789/0 conn(0x7f6ec809cb20 legacy=0x7f6ec80ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6ec6ffd640 1 -- 172.21.15.28:0/334283688 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 178201044 0 0) 0x7f6ec80d1d30 con 0x7f6ec809cb20 2024-04-25T19:12:26.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.487+0000 7f6eccd8b640 1 -- 172.21.15.28:0/334283688 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec806e5f0 con 0x7f6ec8163850 2024-04-25T19:12:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.488+0000 7f6eccd8b640 1 --2- 172.21.15.28:0/334283688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec809bae0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6ec806ed80 tx=0x7f6ec002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449f3d57526c8d22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.488+0000 7f6ec6ffd640 1 -- 172.21.15.28:0/334283688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0007740 con 0x7f6ec8163850 2024-04-25T19:12:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.488+0000 7f6ec6ffd640 1 -- 172.21.15.28:0/334283688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec00078e0 con 0x7f6ec8163850 2024-04-25T19:12:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.488+0000 7f6ec6ffd640 1 -- 172.21.15.28:0/334283688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0007cb0 con 0x7f6ec8163850 2024-04-25T19:12:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.489+0000 7f6ece7ee640 1 -- 172.21.15.28:0/334283688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 msgr2=0x7f6ec809bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.489+0000 7f6ece7ee640 1 --2- 172.21.15.28:0/334283688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec809bae0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6ec806ed80 tx=0x7f6ec002a6f0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.489+0000 7f6ece7ee640 1 -- 172.21.15.28:0/334283688 shutdown_connections 2024-04-25T19:12:26.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.489+0000 7f6ece7ee640 1 --2- 172.21.15.28:0/334283688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec809bae0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.489+0000 7f6ece7ee640 1 -- 172.21.15.28:0/334283688 >> 172.21.15.28:0/334283688 conn(0x7f6ec80b54b0 msgr2=0x7f6ec80b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.489+0000 7f6ece7ee640 1 -- 172.21.15.28:0/334283688 shutdown_connections 2024-04-25T19:12:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.489+0000 7f6ece7ee640 1 -- 172.21.15.28:0/334283688 wait complete. 2024-04-25T19:12:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.491+0000 7f6ece7ee640 1 Processor -- start 2024-04-25T19:12:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.491+0000 7f6ece7ee640 1 -- start start 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6ece7ee640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ec809cb20 0x7f6ec81fa330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6ece7ee640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ec80b91a0 0x7f6ec81565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6ece7ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec80b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6ece7ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec80d22b0 con 0x7f6ec8163850 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6ece7ee640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec80d1fd0 con 0x7f6ec80b91a0 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6ece7ee640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6ec8163c60 con 0x7f6ec809cb20 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6eccd8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec80b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6eccd8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec80b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38376/0 (socket says 172.21.15.28:38376) 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.492+0000 7f6eccd8b640 1 -- 172.21.15.28:0/471868182 learned_addr learned my addr 172.21.15.28:0/471868182 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6ec77fe640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ec80b91a0 0x7f6ec81565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ec809cb20 0x7f6ec81fa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6eccd8b640 1 -- 172.21.15.28:0/471868182 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ec809cb20 msgr2=0x7f6ec81fa330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6eccd8b640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ec809cb20 0x7f6ec81fa330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6eccd8b640 1 -- 172.21.15.28:0/471868182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ec80b91a0 msgr2=0x7f6ec81565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6eccd8b640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ec80b91a0 0x7f6ec81565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6eccd8b640 1 -- 172.21.15.28:0/471868182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec809df20 con 0x7f6ec8163850 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6eccd8b640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec80b4140 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009e50 tx=0x7f6ec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6ec4ff9640 1 -- 172.21.15.28:0/471868182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec003f070 con 0x7f6ec8163850 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6ec4ff9640 1 -- 172.21.15.28:0/471868182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ec0031bf0 con 0x7f6ec8163850 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec809e200 con 0x7f6ec8163850 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6ec4ff9640 1 -- 172.21.15.28:0/471868182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0044460 con 0x7f6ec8163850 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.493+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec809fe80 con 0x7f6ec8163850 2024-04-25T19:12:26.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.494+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ec809dc90 con 0x7f6ec8163850 2024-04-25T19:12:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.495+0000 7f6ec4ff9640 1 -- 172.21.15.28:0/471868182 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6ec003b410 con 0x7f6ec8163850 2024-04-25T19:12:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.496+0000 7f6ec4ff9640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6eb40744b0 0x7f6eb40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.496+0000 7f6ec4ff9640 1 -- 172.21.15.28:0/471868182 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6ec00b6f30 con 0x7f6ec8163850 2024-04-25T19:12:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.496+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6eb40744b0 0x7f6eb40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.496+0000 7f6ec7fff640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6eb40744b0 0x7f6eb40769c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8000f60 tx=0x7f6eb8007e40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:26.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.500+0000 7f6ec4ff9640 1 -- 172.21.15.28:0/471868182 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ec0080bd0 con 0x7f6ec8163850 2024-04-25T19:12:26.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.688+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ec809fc90 con 0x7f6ec8163850 2024-04-25T19:12:26.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.689+0000 7f6ec4ff9640 1 -- 172.21.15.28:0/471868182 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f6ec0084ed0 con 0x7f6ec8163850 2024-04-25T19:12:26.691 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:26.691 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6eb40744b0 msgr2=0x7f6eb40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6eb40744b0 0x7f6eb40769c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8000f60 tx=0x7f6eb8007e40 comp rx=0 tx=0).stop 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 msgr2=0x7f6ec80b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec80b4140 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6ec0009e50 tx=0x7f6ec0004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 shutdown_connections 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6eb40744b0 0x7f6eb40769c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec8163850 0x7f6ec80b4140 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ec80b91a0 0x7f6ec81565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 --2- 172.21.15.28:0/471868182 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6ec809cb20 0x7f6ec81fa330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 >> 172.21.15.28:0/471868182 conn(0x7f6ec80b54b0 msgr2=0x7f6ec80b10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 shutdown_connections 2024-04-25T19:12:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.691+0000 7f6ece7ee640 1 -- 172.21.15.28:0/471868182 wait complete. 2024-04-25T19:12:26.705 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:26.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.818+0000 7f423f8aa640 1 Processor -- start 2024-04-25T19:12:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.818+0000 7f423f8aa640 1 -- start start 2024-04-25T19:12:26.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423f8aa640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42380affc0 0x7f4238167d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423f8aa640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42380b5730 con 0x7f42380d0ee0 2024-04-25T19:12:26.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423f8aa640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f423806f5b0 con 0x7f423815aaa0 2024-04-25T19:12:26.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423f8aa640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f42380af290 con 0x7f42380affc0 2024-04-25T19:12:26.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423ce45640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f42380d0ee0 0x7f42380b5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36940/0 (socket says 172.21.15.28:36940) 2024-04-25T19:12:26.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423ce45640 1 -- 172.21.15.28:0/729969045 learned_addr learned my addr 172.21.15.28:0/729969045 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:26.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423de47640 1 --2- 172.21.15.28:0/729969045 >> v2:172.21.15.53:3300/0 conn(0x7f42380affc0 0x7f4238167d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f4237fff640 1 -- 172.21.15.28:0/729969045 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1350331262 0 0) 0x7f422c001110 con 0x7f42380d0ee0 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f4237fff640 1 -- 172.21.15.28:0/729969045 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4220003520 con 0x7f42380d0ee0 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423de47640 1 -- 172.21.15.28:0/729969045 >> v1:172.21.15.53:6789/0 conn(0x7f423815aaa0 legacy=0x7f42380b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423de47640 1 -- 172.21.15.28:0/729969045 >> v1:172.21.15.28:6789/0 conn(0x7f42380d0ee0 legacy=0x7f42380b5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.820+0000 7f423de47640 1 -- 172.21.15.28:0/729969045 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42380d14a0 con 0x7f42380affc0 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.821+0000 7f423de47640 1 --2- 172.21.15.28:0/729969045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42380affc0 0x7f4238167d20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4228009d90 tx=0x7f422802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e67562a8af90566 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.821+0000 7f4237fff640 1 -- 172.21.15.28:0/729969045 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228007740 con 0x7f42380affc0 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.821+0000 7f4237fff640 1 -- 172.21.15.28:0/729969045 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42280078e0 con 0x7f42380affc0 2024-04-25T19:12:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.821+0000 7f4237fff640 1 -- 172.21.15.28:0/729969045 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4228007c90 con 0x7f42380affc0 2024-04-25T19:12:26.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.822+0000 7f423f8aa640 1 -- 172.21.15.28:0/729969045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42380affc0 msgr2=0x7f4238167d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.822+0000 7f423f8aa640 1 --2- 172.21.15.28:0/729969045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42380affc0 0x7f4238167d20 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4228009d90 tx=0x7f422802a660 comp rx=0 tx=0).stop 2024-04-25T19:12:26.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.822+0000 7f423f8aa640 1 -- 172.21.15.28:0/729969045 shutdown_connections 2024-04-25T19:12:26.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.822+0000 7f423f8aa640 1 --2- 172.21.15.28:0/729969045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42380affc0 0x7f4238167d20 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.822+0000 7f423f8aa640 1 -- 172.21.15.28:0/729969045 >> 172.21.15.28:0/729969045 conn(0x7f42380742c0 msgr2=0x7f423806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:26.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.822+0000 7f423f8aa640 1 -- 172.21.15.28:0/729969045 shutdown_connections 2024-04-25T19:12:26.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.822+0000 7f423f8aa640 1 -- 172.21.15.28:0/729969045 wait complete. 2024-04-25T19:12:26.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.824+0000 7f423f8aa640 1 Processor -- start 2024-04-25T19:12:26.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.824+0000 7f423f8aa640 1 -- start start 2024-04-25T19:12:26.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423f8aa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42380affc0 0x7f423814ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423f8aa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42380d0ee0 0x7f423813e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423f8aa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f423815aaa0 0x7f4238150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423f8aa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42380afaa0 con 0x7f42380affc0 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423f8aa640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f42380af7c0 con 0x7f423815aaa0 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423f8aa640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4238168260 con 0x7f42380d0ee0 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423d646640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42380affc0 0x7f423814ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423d646640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42380affc0 0x7f423814ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38382/0 (socket says 172.21.15.28:38382) 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423d646640 1 -- 172.21.15.28:0/3471958200 learned_addr learned my addr 172.21.15.28:0/3471958200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423de47640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f423815aaa0 0x7f4238150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.825+0000 7f423ce45640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42380d0ee0 0x7f423813e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f423d646640 1 -- 172.21.15.28:0/3471958200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42380d0ee0 msgr2=0x7f423813e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f423d646640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42380d0ee0 0x7f423813e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f423d646640 1 -- 172.21.15.28:0/3471958200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f423815aaa0 msgr2=0x7f4238150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f423d646640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f423815aaa0 0x7f4238150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f423d646640 1 -- 172.21.15.28:0/3471958200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f423814c0f0 con 0x7f42380affc0 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f423d646640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42380affc0 0x7f423814ceb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f423000c3f0 tx=0x7f423000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f4235ffb640 1 -- 172.21.15.28:0/3471958200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42300052d0 con 0x7f42380affc0 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f4235ffb640 1 -- 172.21.15.28:0/3471958200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4230005490 con 0x7f42380affc0 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f4235ffb640 1 -- 172.21.15.28:0/3471958200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4230007260 con 0x7f42380affc0 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f423814c3d0 con 0x7f42380affc0 2024-04-25T19:12:26.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.826+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f423814c990 con 0x7f42380affc0 2024-04-25T19:12:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.827+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4238050740 con 0x7f42380affc0 2024-04-25T19:12:26.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.828+0000 7f4235ffb640 1 -- 172.21.15.28:0/3471958200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4230007400 con 0x7f42380affc0 2024-04-25T19:12:26.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.828+0000 7f4235ffb640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f42240744b0 0x7f42240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:26.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.828+0000 7f4235ffb640 1 -- 172.21.15.28:0/3471958200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4230094920 con 0x7f42380affc0 2024-04-25T19:12:26.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.828+0000 7f423ce45640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f42240744b0 0x7f42240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:26.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.829+0000 7f423ce45640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f42240744b0 0x7f42240769c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f423806dbc0 tx=0x7f422c007b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:26.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:26.833+0000 7f4235ffb640 1 -- 172.21.15.28:0/3471958200 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f423005e6e0 con 0x7f42380affc0 2024-04-25T19:12:27.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.020+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f42380cdba0 con 0x7f42380affc0 2024-04-25T19:12:27.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.020+0000 7f4235ffb640 1 -- 172.21.15.28:0/3471958200 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f42300629e0 con 0x7f42380affc0 2024-04-25T19:12:27.022 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:27.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f42240744b0 msgr2=0x7f42240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f42240744b0 0x7f42240769c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f423806dbc0 tx=0x7f422c007b70 comp rx=0 tx=0).stop 2024-04-25T19:12:27.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42380affc0 msgr2=0x7f423814ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42380affc0 0x7f423814ceb0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f423000c3f0 tx=0x7f423000c8b0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 shutdown_connections 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f42240744b0 0x7f42240769c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f423815aaa0 0x7f4238150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f42380d0ee0 0x7f423813e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 --2- 172.21.15.28:0/3471958200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42380affc0 0x7f423814ceb0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 >> 172.21.15.28:0/3471958200 conn(0x7f42380742c0 msgr2=0x7f423806cb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 shutdown_connections 2024-04-25T19:12:27.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.022+0000 7f423f8aa640 1 -- 172.21.15.28:0/3471958200 wait complete. 2024-04-25T19:12:27.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.149+0000 7f0d63005640 1 Processor -- start 2024-04-25T19:12:27.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.149+0000 7f0d63005640 1 -- start start 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d63005640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c160250 0x7f0d5c0b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d63005640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d5c164b20 con 0x7f0d5c0d2ca0 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d63005640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d5c163d60 con 0x7f0d5c0b81a0 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d63005640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5c0b4f50 con 0x7f0d5c160250 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d615a2640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0d5c0d2ca0 0x7f0d5c0d44b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36944/0 (socket says 172.21.15.28:36944) 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d615a2640 1 -- 172.21.15.28:0/752726559 learned_addr learned my addr 172.21.15.28:0/752726559 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d60da1640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c160250 0x7f0d5c0b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1511149365 0 0) 0x7f0d5c164b20 con 0x7f0d5c0d2ca0 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.150+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d48003520 con 0x7f0d5c0d2ca0 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1887710397 0 0) 0x7f0d5c163d60 con 0x7f0d5c0b81a0 2024-04-25T19:12:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d5c164b20 con 0x7f0d5c0b81a0 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 434590641 0 0) 0x7f0d48003520 con 0x7f0d5c0d2ca0 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c160250 msgr2=0x7f0d5c0b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c160250 0x7f0d5c0b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 >> v1:172.21.15.53:6789/0 conn(0x7f0d5c0b81a0 legacy=0x7f0d5c0d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5c164040 con 0x7f0d5c0d2ca0 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f0d4c001670 con 0x7f0d5c0d2ca0 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 >> v1:172.21.15.28:6789/0 conn(0x7f0d5c0d2ca0 legacy=0x7f0d5c0d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d48003b70 0x7f0d48003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c160250 0x7f0d48004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48004d40 0x7f0d48005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c0b4f50 con 0x7f0d48003b70 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4800bb80 con 0x7f0d48004d40 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0d4800bd80 con 0x7f0d5c160250 2024-04-25T19:12:27.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d60da1640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d48003b70 0x7f0d48003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d615a2640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48004d40 0x7f0d48005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d5bfff640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c160250 0x7f0d48004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d60da1640 1 -- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c160250 msgr2=0x7f0d48004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d60da1640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c160250 0x7f0d48004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d60da1640 1 -- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48004d40 msgr2=0x7f0d48005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d60da1640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48004d40 0x7f0d48005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d60da1640 1 -- 172.21.15.28:0/752726559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d48003520 con 0x7f0d48003b70 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.151+0000 7f0d615a2640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48004d40 0x7f0d48005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.152+0000 7f0d60da1640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d48003b70 0x7f0d48003fc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009f90 tx=0x7f0d48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647b98f9cb0393ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.152+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54009c10 con 0x7f0d48003b70 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.152+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d54036070 con 0x7f0d48003b70 2024-04-25T19:12:27.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.152+0000 7f0d5b7fe640 1 -- 172.21.15.28:0/752726559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54031810 con 0x7f0d48003b70 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.153+0000 7f0d63005640 1 -- 172.21.15.28:0/752726559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d48003b70 msgr2=0x7f0d48003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.153+0000 7f0d63005640 1 --2- 172.21.15.28:0/752726559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d48003b70 0x7f0d48003fc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0d54009f90 tx=0x7f0d48005830 comp rx=0 tx=0).stop 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.153+0000 7f0d615a2640 1 -- 172.21.15.28:0/752726559 reap_dead start 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.153+0000 7f0d63005640 1 -- 172.21.15.28:0/752726559 shutdown_connections 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.153+0000 7f0d63005640 1 -- 172.21.15.28:0/752726559 >> 172.21.15.28:0/752726559 conn(0x7f0d5c06ee30 msgr2=0x7f0d5c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.153+0000 7f0d63005640 1 -- 172.21.15.28:0/752726559 shutdown_connections 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.153+0000 7f0d63005640 1 -- 172.21.15.28:0/752726559 wait complete. 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.155+0000 7f0d63005640 1 Processor -- start 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.155+0000 7f0d63005640 1 -- start start 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d63005640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48003b70 0x7f0d5c15ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d63005640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c1f9b30 0x7f0d5c1f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d63005640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1fa480 0x7f0d5c0ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d63005640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c0b4690 con 0x7f0d5c1fa480 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d63005640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c164600 con 0x7f0d48003b70 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d63005640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5c0b4810 con 0x7f0d5c1f9b30 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d615a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1fa480 0x7f0d5c0ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d615a2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1fa480 0x7f0d5c0ae0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38400/0 (socket says 172.21.15.28:38400) 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d615a2640 1 -- 172.21.15.28:0/943226616 learned_addr learned my addr 172.21.15.28:0/943226616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d60da1640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48003b70 0x7f0d5c15ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.156+0000 7f0d5bfff640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c1f9b30 0x7f0d5c1f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d615a2640 1 -- 172.21.15.28:0/943226616 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c1f9b30 msgr2=0x7f0d5c1f9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d615a2640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c1f9b30 0x7f0d5c1f9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d615a2640 1 -- 172.21.15.28:0/943226616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48003b70 msgr2=0x7f0d5c15ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d615a2640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48003b70 0x7f0d5c15ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d615a2640 1 -- 172.21.15.28:0/943226616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5c0ad400 con 0x7f0d5c1fa480 2024-04-25T19:12:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d615a2640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1fa480 0x7f0d5c0ae0a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000ff0 tx=0x7f0d4c00e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d597fa640 1 -- 172.21.15.28:0/943226616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c019070 con 0x7f0d5c1fa480 2024-04-25T19:12:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d597fa640 1 -- 172.21.15.28:0/943226616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d4c010510 con 0x7f0d5c1fa480 2024-04-25T19:12:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5c0ad6e0 con 0x7f0d5c1fa480 2024-04-25T19:12:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d597fa640 1 -- 172.21.15.28:0/943226616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0108e0 con 0x7f0d5c1fa480 2024-04-25T19:12:27.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.157+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d5c0ccab0 con 0x7f0d5c1fa480 2024-04-25T19:12:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.158+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d5c050740 con 0x7f0d5c1fa480 2024-04-25T19:12:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.159+0000 7f0d597fa640 1 -- 172.21.15.28:0/943226616 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0d4c010a80 con 0x7f0d5c1fa480 2024-04-25T19:12:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.159+0000 7f0d597fa640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d3c0744b0 0x7f0d3c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.159+0000 7f0d60da1640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d3c0744b0 0x7f0d3c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.159+0000 7f0d597fa640 1 -- 172.21.15.28:0/943226616 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0d4c094a30 con 0x7f0d5c1fa480 2024-04-25T19:12:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.160+0000 7f0d60da1640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d3c0744b0 0x7f0d3c0769c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0d54035000 tx=0x7f0d5402f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.164+0000 7f0d597fa640 1 -- 172.21.15.28:0/943226616 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d4c05e6f0 con 0x7f0d5c1fa480 2024-04-25T19:12:27.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.350+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0d5c06c570 con 0x7f0d5c1fa480 2024-04-25T19:12:27.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.350+0000 7f0d597fa640 1 -- 172.21.15.28:0/943226616 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f0d4c0629f0 con 0x7f0d5c1fa480 2024-04-25T19:12:27.352 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d3c0744b0 msgr2=0x7f0d3c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d3c0744b0 0x7f0d3c0769c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0d54035000 tx=0x7f0d5402f000 comp rx=0 tx=0).stop 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1fa480 msgr2=0x7f0d5c0ae0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1fa480 0x7f0d5c0ae0a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c000ff0 tx=0x7f0d4c00e890 comp rx=0 tx=0).stop 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 shutdown_connections 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0d3c0744b0 0x7f0d3c0769c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d5c1fa480 0x7f0d5c0ae0a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0d5c1f9b30 0x7f0d5c1f9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 --2- 172.21.15.28:0/943226616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d48003b70 0x7f0d5c15ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 >> 172.21.15.28:0/943226616 conn(0x7f0d5c06ee30 msgr2=0x7f0d5c0d5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 shutdown_connections 2024-04-25T19:12:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.352+0000 7f0d63005640 1 -- 172.21.15.28:0/943226616 wait complete. 2024-04-25T19:12:27.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.478+0000 7f34b5f96640 1 Processor -- start 2024-04-25T19:12:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.478+0000 7f34b5f96640 1 -- start start 2024-04-25T19:12:27.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.479+0000 7f34b5f96640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34b009caf0 0x7f34b00d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.479+0000 7f34b5f96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009cf00 0x7f34b00ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.479+0000 7f34b5f96640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b00d1ce0 con 0x7f34b00d2670 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.479+0000 7f34b5f96640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f34b00b8fa0 con 0x7f34b009caf0 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.479+0000 7f34b5f96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f34b00cf8d0 con 0x7f34b009cf00 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.479+0000 7f34af7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34b009caf0 0x7f34b00d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.479+0000 7f34af7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34b009caf0 0x7f34b00d2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38410/0 (socket says 172.21.15.28:38410) 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.479+0000 7f34af7fe640 1 -- 172.21.15.28:0/2486778789 learned_addr learned my addr 172.21.15.28:0/2486778789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.480+0000 7f34affff640 1 --2- 172.21.15.28:0/2486778789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009cf00 0x7f34b00ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.480+0000 7f34af7fe640 1 -- 172.21.15.28:0/2486778789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009cf00 msgr2=0x7f34b00ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.480+0000 7f34af7fe640 1 --2- 172.21.15.28:0/2486778789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009cf00 0x7f34b00ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.480+0000 7f34af7fe640 1 -- 172.21.15.28:0/2486778789 >> v1:172.21.15.53:6789/0 conn(0x7f34b00d2670 legacy=0x7f34b00b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.480+0000 7f34af7fe640 1 -- 172.21.15.28:0/2486778789 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b00cfbb0 con 0x7f34b009caf0 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.480+0000 7f34affff640 1 --2- 172.21.15.28:0/2486778789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009cf00 0x7f34b00ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:27.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.480+0000 7f34af7fe640 1 --2- 172.21.15.28:0/2486778789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009caf0 0x7f34b00d2130 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009870 tx=0x7f34a002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9418ac19ef3570f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34ae7fc640 1 -- 172.21.15.28:0/2486778789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a0037070 con 0x7f34b009caf0 2024-04-25T19:12:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34ae7fc640 1 -- 172.21.15.28:0/2486778789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a0032440 con 0x7f34b009caf0 2024-04-25T19:12:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34ae7fc640 1 -- 172.21.15.28:0/2486778789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a0032810 con 0x7f34b009caf0 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34b5f96640 1 -- 172.21.15.28:0/2486778789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009caf0 msgr2=0x7f34b00d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34b5f96640 1 --2- 172.21.15.28:0/2486778789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009caf0 0x7f34b00d2130 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f34a0009870 tx=0x7f34a002a1c0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34b5f96640 1 -- 172.21.15.28:0/2486778789 shutdown_connections 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34b5f96640 1 --2- 172.21.15.28:0/2486778789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009cf00 0x7f34b00ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34b5f96640 1 --2- 172.21.15.28:0/2486778789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009caf0 0x7f34b00d2130 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34b5f96640 1 -- 172.21.15.28:0/2486778789 >> 172.21.15.28:0/2486778789 conn(0x7f34b00d03f0 msgr2=0x7f34b00d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34b5f96640 1 -- 172.21.15.28:0/2486778789 shutdown_connections 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.481+0000 7f34b5f96640 1 -- 172.21.15.28:0/2486778789 wait complete. 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.483+0000 7f34b5f96640 1 Processor -- start 2024-04-25T19:12:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.483+0000 7f34b5f96640 1 -- start start 2024-04-25T19:12:27.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34b5f96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009caf0 0x7f34b01dd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34b5f96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009cf00 0x7f34b01dc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34b5f96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b00d2670 0x7f34b00cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34b5f96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34b00b9430 con 0x7f34b009cf00 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34b5f96640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f34b00b78e0 con 0x7f34b00d2670 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34b5f96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f34b00d4790 con 0x7f34b009caf0 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009cf00 0x7f34b01dc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009cf00 0x7f34b01dc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38412/0 (socket says 172.21.15.28:38412) 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 -- 172.21.15.28:0/3819215943 learned_addr learned my addr 172.21.15.28:0/3819215943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34af7fe640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009caf0 0x7f34b01dd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34affff640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b00d2670 0x7f34b00cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 -- 172.21.15.28:0/3819215943 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009caf0 msgr2=0x7f34b01dd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009caf0 0x7f34b01dd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 -- 172.21.15.28:0/3819215943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b00d2670 msgr2=0x7f34b00cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b00d2670 0x7f34b00cc1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 -- 172.21.15.28:0/3819215943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b01de340 con 0x7f34b009cf00 2024-04-25T19:12:27.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.485+0000 7f34aeffd640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009cf00 0x7f34b01dc7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c3f0 tx=0x7f34a400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.486+0000 7f3497fff640 1 -- 172.21.15.28:0/3819215943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a40052d0 con 0x7f34b009cf00 2024-04-25T19:12:27.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.486+0000 7f3497fff640 1 -- 172.21.15.28:0/3819215943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a4005490 con 0x7f34b009cf00 2024-04-25T19:12:27.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.486+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b00c8650 con 0x7f34b009cf00 2024-04-25T19:12:27.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.486+0000 7f3497fff640 1 -- 172.21.15.28:0/3819215943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a4007260 con 0x7f34b009cf00 2024-04-25T19:12:27.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.486+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b00c8c10 con 0x7f34b009cf00 2024-04-25T19:12:27.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.487+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b0050740 con 0x7f34b009cf00 2024-04-25T19:12:27.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.487+0000 7f3497fff640 1 -- 172.21.15.28:0/3819215943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f34a4007400 con 0x7f34b009cf00 2024-04-25T19:12:27.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.488+0000 7f3497fff640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f349c0744b0 0x7f349c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.488+0000 7f34af7fe640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f349c0744b0 0x7f349c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.488+0000 7f3497fff640 1 -- 172.21.15.28:0/3819215943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f34a4094920 con 0x7f34b009cf00 2024-04-25T19:12:27.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.488+0000 7f34af7fe640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f349c0744b0 0x7f349c0769c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f34a0034000 tx=0x7f34a002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.493+0000 7f3497fff640 1 -- 172.21.15.28:0/3819215943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34a405e6f0 con 0x7f34b009cf00 2024-04-25T19:12:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.678+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f34b00717d0 con 0x7f34b009cf00 2024-04-25T19:12:27.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.679+0000 7f3497fff640 1 -- 172.21.15.28:0/3819215943 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f34a40629f0 con 0x7f34b009cf00 2024-04-25T19:12:27.681 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f349c0744b0 msgr2=0x7f349c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f349c0744b0 0x7f349c0769c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f34a0034000 tx=0x7f34a002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009cf00 msgr2=0x7f34b01dc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009cf00 0x7f34b01dc7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c3f0 tx=0x7f34a400c8b0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 shutdown_connections 2024-04-25T19:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f349c0744b0 0x7f349c0769c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b00d2670 0x7f34b00cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b009cf00 0x7f34b01dc7a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 --2- 172.21.15.28:0/3819215943 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f34b009caf0 0x7f34b01dd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 >> 172.21.15.28:0/3819215943 conn(0x7f34b00d03f0 msgr2=0x7f34b0072d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 shutdown_connections 2024-04-25T19:12:27.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.681+0000 7f34b5f96640 1 -- 172.21.15.28:0/3819215943 wait complete. 2024-04-25T19:12:27.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.807+0000 7f64781be640 1 Processor -- start 2024-04-25T19:12:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.807+0000 7f64781be640 1 -- start start 2024-04-25T19:12:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f64781be640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f647009cd90 0x7f647006e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f64781be640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64700b39e0 0x7f64700d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f64781be640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f647009d1a0 con 0x7f647006e840 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f64781be640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f647006df50 con 0x7f64700b39e0 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f64781be640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f64700d4870 con 0x7f647009cd90 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f647675b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64700b39e0 0x7f64700d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f647675b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f64700b39e0 0x7f64700d1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38418/0 (socket says 172.21.15.28:38418) 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f647675b640 1 -- 172.21.15.28:0/1601617904 learned_addr learned my addr 172.21.15.28:0/1601617904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.808+0000 7f6475f5a640 1 --2- 172.21.15.28:0/1601617904 >> v2:172.21.15.53:3300/0 conn(0x7f647009cd90 0x7f647006e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.809+0000 7f647675b640 1 -- 172.21.15.28:0/1601617904 >> v2:172.21.15.53:3300/0 conn(0x7f647009cd90 msgr2=0x7f647006e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.809+0000 7f647675b640 1 --2- 172.21.15.28:0/1601617904 >> v2:172.21.15.53:3300/0 conn(0x7f647009cd90 0x7f647006e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.809+0000 7f647675b640 1 -- 172.21.15.28:0/1601617904 >> v1:172.21.15.53:6789/0 conn(0x7f647006e840 legacy=0x7f64700b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.809+0000 7f647675b640 1 -- 172.21.15.28:0/1601617904 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647006d960 con 0x7f64700b39e0 2024-04-25T19:12:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.809+0000 7f647675b640 1 --2- 172.21.15.28:0/1601617904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64700b39e0 0x7f64700d1e50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f647008cd70 tx=0x7f646002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492818a4aa349eb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f6474f58640 1 -- 172.21.15.28:0/1601617904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460036070 con 0x7f64700b39e0 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f6474f58640 1 -- 172.21.15.28:0/1601617904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6460031600 con 0x7f64700b39e0 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f6474f58640 1 -- 172.21.15.28:0/1601617904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64600319d0 con 0x7f64700b39e0 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f64781be640 1 -- 172.21.15.28:0/1601617904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64700b39e0 msgr2=0x7f64700d1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f64781be640 1 --2- 172.21.15.28:0/1601617904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64700b39e0 0x7f64700d1e50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f647008cd70 tx=0x7f646002a6a0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f64781be640 1 -- 172.21.15.28:0/1601617904 shutdown_connections 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f64781be640 1 --2- 172.21.15.28:0/1601617904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64700b39e0 0x7f64700d1e50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f64781be640 1 --2- 172.21.15.28:0/1601617904 >> v2:172.21.15.53:3300/0 conn(0x7f647009cd90 0x7f647006e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f64781be640 1 -- 172.21.15.28:0/1601617904 >> 172.21.15.28:0/1601617904 conn(0x7f64700d03f0 msgr2=0x7f64700d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f64781be640 1 -- 172.21.15.28:0/1601617904 shutdown_connections 2024-04-25T19:12:27.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.810+0000 7f64781be640 1 -- 172.21.15.28:0/1601617904 wait complete. 2024-04-25T19:12:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.812+0000 7f64781be640 1 Processor -- start 2024-04-25T19:12:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.812+0000 7f64781be640 1 -- start start 2024-04-25T19:12:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.813+0000 7f64781be640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f647006e840 0x7f64700cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f64781be640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647009cd90 0x7f647009fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f64781be640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64700b39e0 0x7f64701de7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f64781be640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64700b9180 con 0x7f647009cd90 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f64781be640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f647006dc70 con 0x7f647006e840 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f64781be640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f64700b9460 con 0x7f64700b39e0 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647009cd90 0x7f647009fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647009cd90 0x7f647009fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38432/0 (socket says 172.21.15.28:38432) 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 -- 172.21.15.28:0/2055796956 learned_addr learned my addr 172.21.15.28:0/2055796956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475f5a640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f647006e840 0x7f64700cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f647675b640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64700b39e0 0x7f64701de7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 -- 172.21.15.28:0/2055796956 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64700b39e0 msgr2=0x7f64701de7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64700b39e0 0x7f64701de7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 -- 172.21.15.28:0/2055796956 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f647006e840 msgr2=0x7f64700cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f647006e840 0x7f64700cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 -- 172.21.15.28:0/2055796956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64700cbe60 con 0x7f647009cd90 2024-04-25T19:12:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f6475759640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647009cd90 0x7f647009fc00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f646400c470 tx=0x7f646400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f646e7fc640 1 -- 172.21.15.28:0/2055796956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64640052d0 con 0x7f647009cd90 2024-04-25T19:12:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f646e7fc640 1 -- 172.21.15.28:0/2055796956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6464005490 con 0x7f647009cd90 2024-04-25T19:12:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f646e7fc640 1 -- 172.21.15.28:0/2055796956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464007260 con 0x7f647009cd90 2024-04-25T19:12:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.814+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64700cc140 con 0x7f647009cd90 2024-04-25T19:12:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.815+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64701dcbe0 con 0x7f647009cd90 2024-04-25T19:12:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.815+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6470050740 con 0x7f647009cd90 2024-04-25T19:12:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.816+0000 7f646e7fc640 1 -- 172.21.15.28:0/2055796956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6464007400 con 0x7f647009cd90 2024-04-25T19:12:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.817+0000 7f646e7fc640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f645c0744b0 0x7f645c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.817+0000 7f646e7fc640 1 -- 172.21.15.28:0/2055796956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6464094980 con 0x7f647009cd90 2024-04-25T19:12:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.817+0000 7f6475f5a640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f645c0744b0 0x7f645c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.817+0000 7f6475f5a640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f645c0744b0 0x7f645c0769c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6468009aa0 tx=0x7f6468009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:27.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:27.821+0000 7f646e7fc640 1 -- 172.21.15.28:0/2055796956 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f646405e740 con 0x7f647009cd90 2024-04-25T19:12:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.009+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f64700b8c00 con 0x7f647009cd90 2024-04-25T19:12:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.009+0000 7f646e7fc640 1 -- 172.21.15.28:0/2055796956 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f6464062a40 con 0x7f647009cd90 2024-04-25T19:12:28.011 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f645c0744b0 msgr2=0x7f645c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f645c0744b0 0x7f645c0769c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6468009aa0 tx=0x7f6468009190 comp rx=0 tx=0).stop 2024-04-25T19:12:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647009cd90 msgr2=0x7f647009fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647009cd90 0x7f647009fc00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f646400c470 tx=0x7f646400c930 comp rx=0 tx=0).stop 2024-04-25T19:12:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 shutdown_connections 2024-04-25T19:12:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f645c0744b0 0x7f645c0769c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64700b39e0 0x7f64701de7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647009cd90 0x7f647009fc00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 --2- 172.21.15.28:0/2055796956 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f647006e840 0x7f64700cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 >> 172.21.15.28:0/2055796956 conn(0x7f64700d03f0 msgr2=0x7f6470072380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 shutdown_connections 2024-04-25T19:12:28.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.011+0000 7f64781be640 1 -- 172.21.15.28:0/2055796956 wait complete. 2024-04-25T19:12:28.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:12:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.135+0000 7f9b70f96640 1 Processor -- start 2024-04-25T19:12:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.135+0000 7f9b70f96640 1 -- start start 2024-04-25T19:12:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b70f96640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9b6c0d1f80 0x7f9b6c15dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b70f96640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b6c0b21c0 0x7f9b6c06ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b70f96640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b6c0b29a0 con 0x7f9b6c161230 2024-04-25T19:12:28.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b70f96640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f9b6c06c570 con 0x7f9b6c0b21c0 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b70f96640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f9b6c0d2540 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b6ad9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b6c0b21c0 0x7f9b6c06ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b6ad9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b6c0b21c0 0x7f9b6c06ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38438/0 (socket says 172.21.15.28:38438) 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b6ad9d640 1 -- 172.21.15.28:0/1482244225 learned_addr learned my addr 172.21.15.28:0/1482244225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b6a59c640 1 --2- 172.21.15.28:0/1482244225 >> v2:172.21.15.53:3300/0 conn(0x7f9b6c0d1f80 0x7f9b6c15dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b6ad9d640 1 -- 172.21.15.28:0/1482244225 >> v2:172.21.15.53:3300/0 conn(0x7f9b6c0d1f80 msgr2=0x7f9b6c15dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b6ad9d640 1 --2- 172.21.15.28:0/1482244225 >> v2:172.21.15.53:3300/0 conn(0x7f9b6c0d1f80 0x7f9b6c15dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b6ad9d640 1 -- 172.21.15.28:0/1482244225 >> v1:172.21.15.53:6789/0 conn(0x7f9b6c161230 legacy=0x7f9b6c0b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:28.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.137+0000 7f9b6ad9d640 1 -- 172.21.15.28:0/1482244225 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c0d2820 con 0x7f9b6c0b21c0 2024-04-25T19:12:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.138+0000 7f9b6ad9d640 1 --2- 172.21.15.28:0/1482244225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0b21c0 0x7f9b6c06ca00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c06fa80 tx=0x7f9b5c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d81ee80849b1290 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.138+0000 7f9b6959a640 1 -- 172.21.15.28:0/1482244225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c036070 con 0x7f9b6c0b21c0 2024-04-25T19:12:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.138+0000 7f9b6959a640 1 -- 172.21.15.28:0/1482244225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b5c031600 con 0x7f9b6c0b21c0 2024-04-25T19:12:28.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.138+0000 7f9b6959a640 1 -- 172.21.15.28:0/1482244225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5c0319d0 con 0x7f9b6c0b21c0 2024-04-25T19:12:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.139+0000 7f9b70f96640 1 -- 172.21.15.28:0/1482244225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0b21c0 msgr2=0x7f9b6c06ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.139+0000 7f9b70f96640 1 --2- 172.21.15.28:0/1482244225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0b21c0 0x7f9b6c06ca00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c06fa80 tx=0x7f9b5c02a660 comp rx=0 tx=0).stop 2024-04-25T19:12:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.139+0000 7f9b70f96640 1 -- 172.21.15.28:0/1482244225 shutdown_connections 2024-04-25T19:12:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.139+0000 7f9b70f96640 1 --2- 172.21.15.28:0/1482244225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0b21c0 0x7f9b6c06ca00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.139+0000 7f9b70f96640 1 --2- 172.21.15.28:0/1482244225 >> v2:172.21.15.53:3300/0 conn(0x7f9b6c0d1f80 0x7f9b6c15dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.139+0000 7f9b70f96640 1 -- 172.21.15.28:0/1482244225 >> 172.21.15.28:0/1482244225 conn(0x7f9b6c073d30 msgr2=0x7f9b6c09b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.139+0000 7f9b70f96640 1 -- 172.21.15.28:0/1482244225 shutdown_connections 2024-04-25T19:12:28.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.139+0000 7f9b70f96640 1 -- 172.21.15.28:0/1482244225 wait complete. 2024-04-25T19:12:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.141+0000 7f9b70f96640 1 Processor -- start 2024-04-25T19:12:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.141+0000 7f9b70f96640 1 -- start start 2024-04-25T19:12:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b70f96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0d1f80 0x7f9b6c149850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b70f96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b6c161230 0x7f9b6c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b70f96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9b6c1559e0 0x7f9b6c1594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b70f96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c161640 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b70f96640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9b6c0d2de0 con 0x7f9b6c161230 2024-04-25T19:12:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b70f96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9b6c06cf40 con 0x7f9b6c1559e0 2024-04-25T19:12:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b6a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0d1f80 0x7f9b6c149850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b6a59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0d1f80 0x7f9b6c149850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38440/0 (socket says 172.21.15.28:38440) 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.142+0000 7f9b6a59c640 1 -- 172.21.15.28:0/3584623497 learned_addr learned my addr 172.21.15.28:0/3584623497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b69d9b640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b6c161230 0x7f9b6c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b6ad9d640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9b6c1559e0 0x7f9b6c1594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b6a59c640 1 -- 172.21.15.28:0/3584623497 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9b6c1559e0 msgr2=0x7f9b6c1594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b6a59c640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9b6c1559e0 0x7f9b6c1594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b6a59c640 1 -- 172.21.15.28:0/3584623497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b6c161230 msgr2=0x7f9b6c158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b6a59c640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b6c161230 0x7f9b6c158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b6a59c640 1 -- 172.21.15.28:0/3584623497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b6c154940 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b6a59c640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0d1f80 0x7f9b6c149850 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400c5f0 tx=0x7f9b6400cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b62ffd640 1 -- 172.21.15.28:0/3584623497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64015070 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b62ffd640 1 -- 172.21.15.28:0/3584623497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b64004220 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b6c154c20 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b62ffd640 1 -- 172.21.15.28:0/3584623497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64003610 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.143+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b6c1551e0 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.145+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b6c050740 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.145+0000 7f9b62ffd640 1 -- 172.21.15.28:0/3584623497 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9b640037b0 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.146+0000 7f9b62ffd640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9b4c0744e0 0x7f9b4c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.146+0000 7f9b69d9b640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9b4c0744e0 0x7f9b4c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.146+0000 7f9b62ffd640 1 -- 172.21.15.28:0/3584623497 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9b64095410 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.146+0000 7f9b69d9b640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9b4c0744e0 0x7f9b4c0769f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9b54006fd0 tx=0x7f9b54008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.151+0000 7f9b62ffd640 1 -- 172.21.15.28:0/3584623497 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b6405f720 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.348+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9b6c09b5e0 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.348+0000 7f9b62ffd640 1 -- 172.21.15.28:0/3584623497 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f9b6405f020 con 0x7f9b6c0d1f80 2024-04-25T19:12:28.350 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-04-25T19:12:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9b4c0744e0 msgr2=0x7f9b4c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9b4c0744e0 0x7f9b4c0769f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9b54006fd0 tx=0x7f9b54008040 comp rx=0 tx=0).stop 2024-04-25T19:12:28.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0d1f80 msgr2=0x7f9b6c149850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0d1f80 0x7f9b6c149850 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400c5f0 tx=0x7f9b6400cab0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 shutdown_connections 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9b4c0744e0 0x7f9b4c0769f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9b6c1559e0 0x7f9b6c1594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b6c161230 0x7f9b6c158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 --2- 172.21.15.28:0/3584623497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b6c0d1f80 0x7f9b6c149850 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 >> 172.21.15.28:0/3584623497 conn(0x7f9b6c073d30 msgr2=0x7f9b6c15a780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.350+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 shutdown_connections 2024-04-25T19:12:28.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.351+0000 7f9b70f96640 1 -- 172.21.15.28:0/3584623497 wait complete. 2024-04-25T19:12:28.363 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-25T19:12:28.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:28.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-25T19:12:28.396 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:12:28.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:28.396 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T19:12:28.455 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T19:12:28.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:28.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.565+0000 7f8ab7906640 1 Processor -- start 2024-04-25T19:12:28.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.565+0000 7f8ab7906640 1 -- start start 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.566+0000 7f8ab7906640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ab00b5130 0x7f8ab0161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.566+0000 7f8ab7906640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8ab006ded0 0x7f8ab006e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.566+0000 7f8ab7906640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ab006e820 con 0x7f8ab00d0830 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.566+0000 7f8ab7906640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8ab0163c30 con 0x7f8ab00b5130 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.566+0000 7f8ab7906640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f8ab0164a90 con 0x7f8ab006ded0 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.567+0000 7f8ab56a2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ab00b5130 0x7f8ab0161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.567+0000 7f8ab56a2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ab00b5130 0x7f8ab0161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38444/0 (socket says 172.21.15.28:38444) 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.567+0000 7f8ab56a2640 1 -- 172.21.15.28:0/1991555608 learned_addr learned my addr 172.21.15.28:0/1991555608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.567+0000 7f8ab56a2640 1 -- 172.21.15.28:0/1991555608 >> v2:172.21.15.53:3300/0 conn(0x7f8ab006ded0 msgr2=0x7f8ab006e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:28.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.567+0000 7f8ab56a2640 1 --2- 172.21.15.28:0/1991555608 >> v2:172.21.15.53:3300/0 conn(0x7f8ab006ded0 0x7f8ab006e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.567+0000 7f8ab56a2640 1 -- 172.21.15.28:0/1991555608 >> v1:172.21.15.53:6789/0 conn(0x7f8ab00d0830 legacy=0x7f8ab00d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:28.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.567+0000 7f8ab56a2640 1 -- 172.21.15.28:0/1991555608 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab0163e80 con 0x7f8ab00b5130 2024-04-25T19:12:28.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.568+0000 7f8ab56a2640 1 --2- 172.21.15.28:0/1991555608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00b5130 0x7f8ab0161130 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8ab008e180 tx=0x7f8aac02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555d40ce651fc55a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.568+0000 7f8aa7fff640 1 -- 172.21.15.28:0/1991555608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac037070 con 0x7f8ab00b5130 2024-04-25T19:12:28.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.568+0000 7f8aa7fff640 1 -- 172.21.15.28:0/1991555608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aac032440 con 0x7f8ab00b5130 2024-04-25T19:12:28.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.568+0000 7f8aa7fff640 1 -- 172.21.15.28:0/1991555608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac032810 con 0x7f8ab00b5130 2024-04-25T19:12:28.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.569+0000 7f8ab7906640 1 -- 172.21.15.28:0/1991555608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00b5130 msgr2=0x7f8ab0161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.569+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1991555608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00b5130 0x7f8ab0161130 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8ab008e180 tx=0x7f8aac02a230 comp rx=0 tx=0).stop 2024-04-25T19:12:28.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.569+0000 7f8ab7906640 1 -- 172.21.15.28:0/1991555608 shutdown_connections 2024-04-25T19:12:28.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.569+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1991555608 >> v2:172.21.15.53:3300/0 conn(0x7f8ab006ded0 0x7f8ab006e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.569+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1991555608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00b5130 0x7f8ab0161130 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.569+0000 7f8ab7906640 1 -- 172.21.15.28:0/1991555608 >> 172.21.15.28:0/1991555608 conn(0x7f8ab00b7ad0 msgr2=0x7f8ab00b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:28.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.569+0000 7f8ab7906640 1 -- 172.21.15.28:0/1991555608 shutdown_connections 2024-04-25T19:12:28.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.569+0000 7f8ab7906640 1 -- 172.21.15.28:0/1991555608 wait complete. 2024-04-25T19:12:28.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.570+0000 7f8ab7906640 1 Processor -- start 2024-04-25T19:12:28.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.571+0000 7f8ab7906640 1 -- start start 2024-04-25T19:12:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab7906640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ab006ded0 0x7f8ab01546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab7906640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab00b5130 0x7f8ab00b69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab7906640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00d0830 0x7f8ab00b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab7906640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab0164720 con 0x7f8ab00d0830 2024-04-25T19:12:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab7906640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab0164440 con 0x7f8ab00b5130 2024-04-25T19:12:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab7906640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8ab01607e0 con 0x7f8ab006ded0 2024-04-25T19:12:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00d0830 0x7f8ab00b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00d0830 0x7f8ab00b6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38452/0 (socket says 172.21.15.28:38452) 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 -- 172.21.15.28:0/1208878769 learned_addr learned my addr 172.21.15.28:0/1208878769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab4ea1640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab00b5130 0x7f8ab00b69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab56a2640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ab006ded0 0x7f8ab01546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 -- 172.21.15.28:0/1208878769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ab006ded0 msgr2=0x7f8ab01546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ab006ded0 0x7f8ab01546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 -- 172.21.15.28:0/1208878769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab00b5130 msgr2=0x7f8ab00b69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab00b5130 0x7f8ab00b69d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 -- 172.21.15.28:0/1208878769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab0156db0 con 0x7f8ab00d0830 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.572+0000 7f8ab5ea3640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00d0830 0x7f8ab00b6f10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800b990 tx=0x7f8aa800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.573+0000 7f8aa5ffb640 1 -- 172.21.15.28:0/1208878769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8019070 con 0x7f8ab00d0830 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.573+0000 7f8aa5ffb640 1 -- 172.21.15.28:0/1208878769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa800cba0 con 0x7f8ab00d0830 2024-04-25T19:12:28.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.573+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab0155d10 con 0x7f8ab00d0830 2024-04-25T19:12:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.573+0000 7f8aa5ffb640 1 -- 172.21.15.28:0/1208878769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8014440 con 0x7f8ab00d0830 2024-04-25T19:12:28.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.573+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ab01562d0 con 0x7f8ab00d0830 2024-04-25T19:12:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.574+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ab0050740 con 0x7f8ab00d0830 2024-04-25T19:12:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.575+0000 7f8aa5ffb640 1 -- 172.21.15.28:0/1208878769 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8aa801c070 con 0x7f8ab00d0830 2024-04-25T19:12:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.575+0000 7f8aa5ffb640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a940744b0 0x7f8a940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.575+0000 7f8aa5ffb640 1 -- 172.21.15.28:0/1208878769 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8aa8095390 con 0x7f8ab00d0830 2024-04-25T19:12:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.575+0000 7f8ab56a2640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a940744b0 0x7f8a940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.576+0000 7f8ab56a2640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a940744b0 0x7f8a940769c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8aac034000 tx=0x7f8aac02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.580+0000 7f8aa5ffb640 1 -- 172.21.15.28:0/1208878769 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8aa805f7b0 con 0x7f8ab00d0830 2024-04-25T19:12:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.807+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ab00d2fe0 con 0x7f8ab00d0830 2024-04-25T19:12:28.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.808+0000 7f8aa5ffb640 1 -- 172.21.15.28:0/1208878769 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f8aa805f0b0 con 0x7f8ab00d0830 2024-04-25T19:12:28.810 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:28.810 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:51.367677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:12:28.810 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a940744b0 msgr2=0x7f8a940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a940744b0 0x7f8a940769c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8aac034000 tx=0x7f8aac02e000 comp rx=0 tx=0).stop 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00d0830 msgr2=0x7f8ab00b6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00d0830 0x7f8ab00b6f10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800b990 tx=0x7f8aa800be50 comp rx=0 tx=0).stop 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 shutdown_connections 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a940744b0 0x7f8a940769c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ab00d0830 0x7f8ab00b6f10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ab00b5130 0x7f8ab00b69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 --2- 172.21.15.28:0/1208878769 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8ab006ded0 0x7f8ab01546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 >> 172.21.15.28:0/1208878769 conn(0x7f8ab00b7ad0 msgr2=0x7f8ab00d4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 shutdown_connections 2024-04-25T19:12:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.810+0000 7f8ab7906640 1 -- 172.21.15.28:0/1208878769 wait complete. 2024-04-25T19:12:28.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.933+0000 7f851e430640 1 Processor -- start 2024-04-25T19:12:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.933+0000 7f851e430640 1 -- start start 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851e430640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f851809cb60 0x7f85180cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851e430640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85180cef50 0x7f85180d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851e430640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f851806d710 con 0x7f851809d010 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851e430640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f851808cc20 con 0x7f85180cef50 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851e430640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f851808ce70 con 0x7f851809cb60 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851c9cd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f85180cef50 0x7f85180d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f8517fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f851809d010 0x7f851806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36948/0 (socket says 172.21.15.28:36948) 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f8517fff640 1 -- 172.21.15.28:0/271972259 learned_addr learned my addr 172.21.15.28:0/271972259 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f85177fe640 1 --2- 172.21.15.28:0/271972259 >> v2:172.21.15.53:3300/0 conn(0x7f851809cb60 0x7f85180cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851c9cd640 1 -- 172.21.15.28:0/271972259 >> v2:172.21.15.53:3300/0 conn(0x7f851809cb60 msgr2=0x7f85180cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851c9cd640 1 --2- 172.21.15.28:0/271972259 >> v2:172.21.15.53:3300/0 conn(0x7f851809cb60 0x7f85180cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851c9cd640 1 -- 172.21.15.28:0/271972259 >> v1:172.21.15.28:6789/0 conn(0x7f851809d010 legacy=0x7f851806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f851c9cd640 1 -- 172.21.15.28:0/271972259 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85180b92d0 con 0x7f85180cef50 2024-04-25T19:12:28.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.935+0000 7f8516ffd640 1 -- 172.21.15.28:0/271972259 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614774988 0 0) 0x7f851806d710 con 0x7f851809d010 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.936+0000 7f851c9cd640 1 --2- 172.21.15.28:0/271972259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85180cef50 0x7f85180d1cb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f851808cda0 tx=0x7f851002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5d5d2817ab78b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.936+0000 7f8516ffd640 1 -- 172.21.15.28:0/271972259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510007740 con 0x7f85180cef50 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.936+0000 7f8516ffd640 1 -- 172.21.15.28:0/271972259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85100078e0 con 0x7f85180cef50 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.936+0000 7f8516ffd640 1 -- 172.21.15.28:0/271972259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8510007cb0 con 0x7f85180cef50 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.937+0000 7f851e430640 1 -- 172.21.15.28:0/271972259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85180cef50 msgr2=0x7f85180d1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.937+0000 7f851e430640 1 --2- 172.21.15.28:0/271972259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85180cef50 0x7f85180d1cb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f851808cda0 tx=0x7f851002a660 comp rx=0 tx=0).stop 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.937+0000 7f851e430640 1 -- 172.21.15.28:0/271972259 shutdown_connections 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.937+0000 7f851e430640 1 --2- 172.21.15.28:0/271972259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85180cef50 0x7f85180d1cb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.937+0000 7f851e430640 1 --2- 172.21.15.28:0/271972259 >> v2:172.21.15.53:3300/0 conn(0x7f851809cb60 0x7f85180cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.937+0000 7f851e430640 1 -- 172.21.15.28:0/271972259 >> 172.21.15.28:0/271972259 conn(0x7f85180d0430 msgr2=0x7f85180d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.937+0000 7f851e430640 1 -- 172.21.15.28:0/271972259 shutdown_connections 2024-04-25T19:12:28.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.937+0000 7f851e430640 1 -- 172.21.15.28:0/271972259 wait complete. 2024-04-25T19:12:28.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.939+0000 7f851e430640 1 Processor -- start 2024-04-25T19:12:28.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.939+0000 7f851e430640 1 -- start start 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f851e430640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851809cb60 0x7f85180c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f851e430640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f851809d010 0x7f85181de4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f851e430640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85180cef50 0x7f8518071fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f851e430640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f851806ea20 con 0x7f851809cb60 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f851e430640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f851806e5e0 con 0x7f851809d010 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f851e430640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f851806eba0 con 0x7f85180cef50 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f8517fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851809cb60 0x7f85180c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f8517fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851809cb60 0x7f85180c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38460/0 (socket says 172.21.15.28:38460) 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f8517fff640 1 -- 172.21.15.28:0/3509419296 learned_addr learned my addr 172.21.15.28:0/3509419296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f85177fe640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f851809d010 0x7f85181de4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f851c9cd640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85180cef50 0x7f8518071fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f8517fff640 1 -- 172.21.15.28:0/3509419296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85180cef50 msgr2=0x7f8518071fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f8517fff640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85180cef50 0x7f8518071fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f8517fff640 1 -- 172.21.15.28:0/3509419296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f851809d010 msgr2=0x7f85181de4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f8517fff640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f851809d010 0x7f85181de4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.940+0000 7f8517fff640 1 -- 172.21.15.28:0/3509419296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85180c9fb0 con 0x7f851809cb60 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.941+0000 7f8517fff640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851809cb60 0x7f85180c96b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f850c00c400 tx=0x7f850c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.941+0000 7f8514ff9640 1 -- 172.21.15.28:0/3509419296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c017070 con 0x7f851809cb60 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.941+0000 7f8514ff9640 1 -- 172.21.15.28:0/3509419296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f850c0074e0 con 0x7f851809cb60 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.941+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85180ca290 con 0x7f851809cb60 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.941+0000 7f8514ff9640 1 -- 172.21.15.28:0/3509419296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f850c0059e0 con 0x7f851809cb60 2024-04-25T19:12:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.941+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85180caf30 con 0x7f851809cb60 2024-04-25T19:12:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.942+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85180ca460 con 0x7f851809cb60 2024-04-25T19:12:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.943+0000 7f8514ff9640 1 -- 172.21.15.28:0/3509419296 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f850c01a070 con 0x7f851809cb60 2024-04-25T19:12:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.943+0000 7f8514ff9640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85040744e0 0x7f85040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.943+0000 7f85177fe640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85040744e0 0x7f85040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.943+0000 7f8514ff9640 1 -- 172.21.15.28:0/3509419296 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f850c094970 con 0x7f851809cb60 2024-04-25T19:12:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.944+0000 7f85177fe640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85040744e0 0x7f85040769f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8518072d40 tx=0x7f8508009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:28.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:28.948+0000 7f8514ff9640 1 -- 172.21.15.28:0/3509419296 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f85180ca460 con 0x7f851809cb60 2024-04-25T19:12:29.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.134+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f85180caa40 con 0x7f851809cb60 2024-04-25T19:12:29.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.135+0000 7f8514ff9640 1 -- 172.21.15.28:0/3509419296 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f850c05e770 con 0x7f851809cb60 2024-04-25T19:12:29.137 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:29.137 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85040744e0 msgr2=0x7f85040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85040744e0 0x7f85040769f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8518072d40 tx=0x7f8508009190 comp rx=0 tx=0).stop 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851809cb60 msgr2=0x7f85180c96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851809cb60 0x7f85180c96b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f850c00c400 tx=0x7f850c00c8c0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 shutdown_connections 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f85040744e0 0x7f85040769f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f85180cef50 0x7f8518071fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f851809d010 0x7f85181de4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 --2- 172.21.15.28:0/3509419296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f851809cb60 0x7f85180c96b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 >> 172.21.15.28:0/3509419296 conn(0x7f85180d0430 msgr2=0x7f851806ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 shutdown_connections 2024-04-25T19:12:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.137+0000 7f851e430640 1 -- 172.21.15.28:0/3509419296 wait complete. 2024-04-25T19:12:29.150 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:11:27.072381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:11:28.257445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T19:11:31.947109+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T19:12:29.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.263+0000 7f86ac757640 1 Processor -- start 2024-04-25T19:12:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.263+0000 7f86ac757640 1 -- start start 2024-04-25T19:12:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86ac757640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a4160250 0x7f86a40b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86ac757640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86a40b81a0 0x7f86a40d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86ac757640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f86a40d2ad0 0x7f86a40d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86ac757640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f86a40b4f50 con 0x7f86a40b81a0 2024-04-25T19:12:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86ac757640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f86a40d2ee0 con 0x7f86a40d2ad0 2024-04-25T19:12:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86ac757640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f86a4164b20 con 0x7f86a4160250 2024-04-25T19:12:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86a9cf2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86a40b81a0 0x7f86a40d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86a9cf2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86a40b81a0 0x7f86a40d2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38464/0 (socket says 172.21.15.28:38464) 2024-04-25T19:12:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86a9cf2640 1 -- 172.21.15.28:0/3869090398 learned_addr learned my addr 172.21.15.28:0/3869090398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86aa4f3640 1 --2- 172.21.15.28:0/3869090398 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a4160250 0x7f86a40b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86aacf4640 1 --2- 172.21.15.28:0/3869090398 >> v2:172.21.15.53:3300/0 conn(0x7f86a40d2ad0 0x7f86a40d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86a9cf2640 1 -- 172.21.15.28:0/3869090398 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a4160250 msgr2=0x7f86a40b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86a9cf2640 1 --2- 172.21.15.28:0/3869090398 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a4160250 0x7f86a40b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86a9cf2640 1 -- 172.21.15.28:0/3869090398 >> v2:172.21.15.53:3300/0 conn(0x7f86a40d2ad0 msgr2=0x7f86a40d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86a9cf2640 1 --2- 172.21.15.28:0/3869090398 >> v2:172.21.15.53:3300/0 conn(0x7f86a40d2ad0 0x7f86a40d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.265+0000 7f86a9cf2640 1 -- 172.21.15.28:0/3869090398 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a4163d60 con 0x7f86a40b81a0 2024-04-25T19:12:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.266+0000 7f86a9cf2640 1 --2- 172.21.15.28:0/3869090398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a40b81a0 0x7f86a40d2590 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f86a40d3890 tx=0x7f869802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688d6ea69882e31e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.266+0000 7f86a94f1640 1 -- 172.21.15.28:0/3869090398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698037070 con 0x7f86a40b81a0 2024-04-25T19:12:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.266+0000 7f86a94f1640 1 -- 172.21.15.28:0/3869090398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8698032440 con 0x7f86a40b81a0 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.266+0000 7f86a94f1640 1 -- 172.21.15.28:0/3869090398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8698032810 con 0x7f86a40b81a0 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 -- 172.21.15.28:0/3869090398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a40b81a0 msgr2=0x7f86a40d2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 --2- 172.21.15.28:0/3869090398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a40b81a0 0x7f86a40d2590 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f86a40d3890 tx=0x7f869802a140 comp rx=0 tx=0).stop 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 -- 172.21.15.28:0/3869090398 shutdown_connections 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 --2- 172.21.15.28:0/3869090398 >> v2:172.21.15.53:3300/0 conn(0x7f86a40d2ad0 0x7f86a40d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 --2- 172.21.15.28:0/3869090398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a40b81a0 0x7f86a40d2590 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 --2- 172.21.15.28:0/3869090398 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a4160250 0x7f86a40b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 -- 172.21.15.28:0/3869090398 >> 172.21.15.28:0/3869090398 conn(0x7f86a406ee30 msgr2=0x7f86a406c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 -- 172.21.15.28:0/3869090398 shutdown_connections 2024-04-25T19:12:29.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.267+0000 7f86ac757640 1 -- 172.21.15.28:0/3869090398 wait complete. 2024-04-25T19:12:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.269+0000 7f86ac757640 1 Processor -- start 2024-04-25T19:12:29.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.269+0000 7f86ac757640 1 -- start start 2024-04-25T19:12:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86ac757640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86a40b81a0 0x7f86a415ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86ac757640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a40d2ad0 0x7f86a40b6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86ac757640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a4160250 0x7f86a40b7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86ac757640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86a4164600 con 0x7f86a4160250 2024-04-25T19:12:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86ac757640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f86a4164320 con 0x7f86a40b81a0 2024-04-25T19:12:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86ac757640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f86a40b4bd0 con 0x7f86a40d2ad0 2024-04-25T19:12:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86aacf4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a4160250 0x7f86a40b7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86aacf4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a4160250 0x7f86a40b7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38478/0 (socket says 172.21.15.28:38478) 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86aacf4640 1 -- 172.21.15.28:0/2681922354 learned_addr learned my addr 172.21.15.28:0/2681922354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86a9cf2640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a40d2ad0 0x7f86a40b6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86aa4f3640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86a40b81a0 0x7f86a415ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.270+0000 7f86aacf4640 1 -- 172.21.15.28:0/2681922354 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a40d2ad0 msgr2=0x7f86a40b6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86aacf4640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a40d2ad0 0x7f86a40b6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86aacf4640 1 -- 172.21.15.28:0/2681922354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86a40b81a0 msgr2=0x7f86a415ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86aacf4640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86a40b81a0 0x7f86a415ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86aacf4640 1 -- 172.21.15.28:0/2681922354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a40cac60 con 0x7f86a4160250 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86aacf4640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a4160250 0x7f86a40b7420 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f869400ef00 tx=0x7f869400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86a2ffd640 1 -- 172.21.15.28:0/2681922354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694018020 con 0x7f86a4160250 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86a2ffd640 1 -- 172.21.15.28:0/2681922354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869400bcc0 con 0x7f86a4160250 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a40caf40 con 0x7f86a4160250 2024-04-25T19:12:29.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86a2ffd640 1 -- 172.21.15.28:0/2681922354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694014640 con 0x7f86a4160250 2024-04-25T19:12:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.271+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a40cb500 con 0x7f86a4160250 2024-04-25T19:12:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.272+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a4050740 con 0x7f86a4160250 2024-04-25T19:12:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.273+0000 7f86a2ffd640 1 -- 172.21.15.28:0/2681922354 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8694005d10 con 0x7f86a4160250 2024-04-25T19:12:29.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.273+0000 7f86a2ffd640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86900744b0 0x7f86900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.273+0000 7f86a2ffd640 1 -- 172.21.15.28:0/2681922354 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8694095a30 con 0x7f86a4160250 2024-04-25T19:12:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.273+0000 7f86aa4f3640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86900744b0 0x7f86900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.274+0000 7f86aa4f3640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86900744b0 0x7f86900769c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f86a40d05f0 tx=0x7f869c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.278+0000 7f86a2ffd640 1 -- 172.21.15.28:0/2681922354 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8694022070 con 0x7f86a4160250 2024-04-25T19:12:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.461+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f86a40ca9f0 con 0x7f86a4160250 2024-04-25T19:12:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.461+0000 7f86a2ffd640 1 -- 172.21.15.28:0/2681922354 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f869405f800 con 0x7f86a4160250 2024-04-25T19:12:29.463 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:29.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86900744b0 msgr2=0x7f86900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86900744b0 0x7f86900769c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f86a40d05f0 tx=0x7f869c009190 comp rx=0 tx=0).stop 2024-04-25T19:12:29.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a4160250 msgr2=0x7f86a40b7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a4160250 0x7f86a40b7420 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f869400ef00 tx=0x7f869400b480 comp rx=0 tx=0).stop 2024-04-25T19:12:29.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 shutdown_connections 2024-04-25T19:12:29.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86900744b0 0x7f86900769c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86a4160250 0x7f86a40b7420 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86a40d2ad0 0x7f86a40b6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 --2- 172.21.15.28:0/2681922354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86a40b81a0 0x7f86a415ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 >> 172.21.15.28:0/2681922354 conn(0x7f86a406ee30 msgr2=0x7f86a415e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:29.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 shutdown_connections 2024-04-25T19:12:29.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.463+0000 7f86ac757640 1 -- 172.21.15.28:0/2681922354 wait complete. 2024-04-25T19:12:29.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.589+0000 7f4886830640 1 Processor -- start 2024-04-25T19:12:29.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.589+0000 7f4886830640 1 -- start start 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f4886830640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f48800b79f0 0x7f48800b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f4886830640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48800b9920 0x7f48800b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f4886830640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4880160720 con 0x7f4880160310 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f4886830640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f48800b8340 con 0x7f48800b79f0 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f4886830640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f48800d0f40 con 0x7f48800b9920 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4880160310 0x7f48800d5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36962/0 (socket says 172.21.15.28:36962) 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487ffff640 1 -- 172.21.15.28:0/4173884375 learned_addr learned my addr 172.21.15.28:0/4173884375 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487f7fe640 1 --2- 172.21.15.28:0/4173884375 >> v2:172.21.15.53:3300/0 conn(0x7f48800b79f0 0x7f48800b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f4884dcd640 1 --2- 172.21.15.28:0/4173884375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48800b9920 0x7f48800b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487effd640 1 -- 172.21.15.28:0/4173884375 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737615494 0 0) 0x7f4880160720 con 0x7f4880160310 2024-04-25T19:12:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487effd640 1 -- 172.21.15.28:0/4173884375 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f486c003520 con 0x7f4880160310 2024-04-25T19:12:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487f7fe640 1 -- 172.21.15.28:0/4173884375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48800b9920 msgr2=0x7f48800b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487f7fe640 1 --2- 172.21.15.28:0/4173884375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48800b9920 0x7f48800b9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487f7fe640 1 -- 172.21.15.28:0/4173884375 >> v1:172.21.15.28:6789/0 conn(0x7f4880160310 legacy=0x7f48800d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.591+0000 7f487f7fe640 1 -- 172.21.15.28:0/4173884375 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48800d1220 con 0x7f48800b79f0 2024-04-25T19:12:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.592+0000 7f487f7fe640 1 --2- 172.21.15.28:0/4173884375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 0x7f48800b7e00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f48680097a0 tx=0x7f486802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e554bd2ca774189 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.592+0000 7f487effd640 1 -- 172.21.15.28:0/4173884375 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4868037070 con 0x7f48800b79f0 2024-04-25T19:12:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.592+0000 7f487effd640 1 -- 172.21.15.28:0/4173884375 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4868032440 con 0x7f48800b79f0 2024-04-25T19:12:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.592+0000 7f487effd640 1 -- 172.21.15.28:0/4173884375 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4868032810 con 0x7f48800b79f0 2024-04-25T19:12:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.593+0000 7f4886830640 1 -- 172.21.15.28:0/4173884375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 msgr2=0x7f48800b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.593+0000 7f4886830640 1 --2- 172.21.15.28:0/4173884375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 0x7f48800b7e00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f48680097a0 tx=0x7f486802a180 comp rx=0 tx=0).stop 2024-04-25T19:12:29.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.593+0000 7f4886830640 1 -- 172.21.15.28:0/4173884375 shutdown_connections 2024-04-25T19:12:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.593+0000 7f4886830640 1 --2- 172.21.15.28:0/4173884375 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f48800b9920 0x7f48800b9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.593+0000 7f4886830640 1 --2- 172.21.15.28:0/4173884375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 0x7f48800b7e00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.593+0000 7f4886830640 1 -- 172.21.15.28:0/4173884375 >> 172.21.15.28:0/4173884375 conn(0x7f48800d2a50 msgr2=0x7f48800d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.593+0000 7f4886830640 1 -- 172.21.15.28:0/4173884375 shutdown_connections 2024-04-25T19:12:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.593+0000 7f4886830640 1 -- 172.21.15.28:0/4173884375 wait complete. 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.595+0000 7f4886830640 1 Processor -- start 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.595+0000 7f4886830640 1 -- start start 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.596+0000 7f4886830640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 0x7f48800d5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.596+0000 7f4886830640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48800b9920 0x7f48800cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.596+0000 7f4886830640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4880160310 0x7f48801f8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.596+0000 7f4886830640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48800d0bf0 con 0x7f48800b9920 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.596+0000 7f4886830640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f48800b9400 con 0x7f48800b79f0 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.596+0000 7f4886830640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f48800cc950 con 0x7f4880160310 2024-04-25T19:12:29.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.596+0000 7f487f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48800b9920 0x7f48800cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48800b9920 0x7f48800cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38486/0 (socket says 172.21.15.28:38486) 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487f7fe640 1 -- 172.21.15.28:0/4050431051 learned_addr learned my addr 172.21.15.28:0/4050431051 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487ffff640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 0x7f48800d5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f4884dcd640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4880160310 0x7f48801f8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487f7fe640 1 -- 172.21.15.28:0/4050431051 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4880160310 msgr2=0x7f48801f8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487f7fe640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4880160310 0x7f48801f8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487f7fe640 1 -- 172.21.15.28:0/4050431051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 msgr2=0x7f48800d5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487f7fe640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 0x7f48800d5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487f7fe640 1 -- 172.21.15.28:0/4050431051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488014d580 con 0x7f48800b9920 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487f7fe640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48800b9920 0x7f48800cef60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4868035040 tx=0x7f48680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487cff9640 1 -- 172.21.15.28:0/4050431051 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4868041030 con 0x7f48800b9920 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487cff9640 1 -- 172.21.15.28:0/4050431051 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f486803d4e0 con 0x7f48800b9920 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488014d860 con 0x7f48800b9920 2024-04-25T19:12:29.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f487cff9640 1 -- 172.21.15.28:0/4050431051 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4868037040 con 0x7f48800b9920 2024-04-25T19:12:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.597+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488014de20 con 0x7f48800b9920 2024-04-25T19:12:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.598+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48800008d0 con 0x7f48800b9920 2024-04-25T19:12:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.599+0000 7f487cff9640 1 -- 172.21.15.28:0/4050431051 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4868044070 con 0x7f48800b9920 2024-04-25T19:12:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.600+0000 7f487cff9640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48600744b0 0x7f48600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.600+0000 7f487ffff640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48600744b0 0x7f48600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.600+0000 7f487cff9640 1 -- 172.21.15.28:0/4050431051 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f48680b5b60 con 0x7f48800b9920 2024-04-25T19:12:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.600+0000 7f487ffff640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48600744b0 0x7f48600769c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4870005880 tx=0x7f4870009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.605+0000 7f487cff9640 1 -- 172.21.15.28:0/4050431051 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f486807f940 con 0x7f48800b9920 2024-04-25T19:12:29.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.788+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f48801f66d0 con 0x7f48800b9920 2024-04-25T19:12:29.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.789+0000 7f487cff9640 1 -- 172.21.15.28:0/4050431051 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f4868083c40 con 0x7f48800b9920 2024-04-25T19:12:29.791 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48600744b0 msgr2=0x7f48600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48600744b0 0x7f48600769c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4870005880 tx=0x7f4870009040 comp rx=0 tx=0).stop 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48800b9920 msgr2=0x7f48800cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48800b9920 0x7f48800cef60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4868035040 tx=0x7f48680046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 shutdown_connections 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48600744b0 0x7f48600769c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4880160310 0x7f48801f8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48800b9920 0x7f48800cef60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 --2- 172.21.15.28:0/4050431051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48800b79f0 0x7f48800d5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 >> 172.21.15.28:0/4050431051 conn(0x7f48800d2a50 msgr2=0x7f48800d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 shutdown_connections 2024-04-25T19:12:29.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.791+0000 7f4886830640 1 -- 172.21.15.28:0/4050431051 wait complete. 2024-04-25T19:12:29.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.917+0000 7f1aa46d9640 1 Processor -- start 2024-04-25T19:12:29.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.917+0000 7f1aa46d9640 1 -- start start 2024-04-25T19:12:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa46d9640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a9c160220 0x7f1a9c164800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa46d9640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a9c0d15b0 con 0x7f1a9c0b4c80 2024-04-25T19:12:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa46d9640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a9c0d12d0 con 0x7f1a9c0b7fd0 2024-04-25T19:12:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa46d9640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f1a9c163d20 con 0x7f1a9c160220 2024-04-25T19:12:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa2475640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1a9c0b4c80 0x7f1a9c0b78c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36972/0 (socket says 172.21.15.28:36972) 2024-04-25T19:12:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa2475640 1 -- 172.21.15.28:0/3323288652 learned_addr learned my addr 172.21.15.28:0/3323288652 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1577223334 0 0) 0x7f1a9c0d15b0 con 0x7f1a9c0b4c80 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a84003520 con 0x7f1a9c0b4c80 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa2c76640 1 --2- 172.21.15.28:0/3323288652 >> v2:172.21.15.53:3300/0 conn(0x7f1a9c160220 0x7f1a9c164800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1079752863 0 0) 0x7f1a84003520 con 0x7f1a9c0b4c80 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 >> v2:172.21.15.53:3300/0 conn(0x7f1a9c160220 msgr2=0x7f1a9c164800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa1473640 1 --2- 172.21.15.28:0/3323288652 >> v2:172.21.15.53:3300/0 conn(0x7f1a9c160220 0x7f1a9c164800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 >> v1:172.21.15.53:6789/0 conn(0x7f1a9c0b7fd0 legacy=0x7f1a9c0d4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.919+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9c164000 con 0x7f1a9c0b4c80 2024-04-25T19:12:29.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f1a94001670 con 0x7f1a9c0b4c80 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 >> v1:172.21.15.28:6789/0 conn(0x7f1a9c0b4c80 legacy=0x7f1a9c0b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1473640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a840039b0 0x7f1a84003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1473640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a84004800 0x7f1a84004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1473640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a840051c0 0x7f1a8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c163d20 con 0x7f1a840051c0 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8400c160 con 0x7f1a840039b0 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f1a8400c360 con 0x7f1a84004800 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa2c76640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a840051c0 0x7f1a8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa2475640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a840039b0 0x7f1a84003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa1c74640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a84004800 0x7f1a84004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa2c76640 1 -- 172.21.15.28:0/3323288652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a84004800 msgr2=0x7f1a84004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa2c76640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a84004800 0x7f1a84004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa2c76640 1 -- 172.21.15.28:0/3323288652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a840039b0 msgr2=0x7f1a84003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:29.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa2c76640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a840039b0 0x7f1a84003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.920+0000 7f1aa2c76640 1 -- 172.21.15.28:0/3323288652 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a84003520 con 0x7f1a840051c0 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.921+0000 7f1aa2c76640 1 -- 172.21.15.28:0/3323288652 reap_dead start 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.921+0000 7f1aa2c76640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a840051c0 0x7f1a8400b9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009fc0 tx=0x7f1a84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866e4c8acd0f54f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.921+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c009c10 con 0x7f1a840051c0 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.921+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a8c003b90 con 0x7f1a840051c0 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.921+0000 7f1aa1473640 1 -- 172.21.15.28:0/3323288652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a8c004020 con 0x7f1a840051c0 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.922+0000 7f1aa46d9640 1 -- 172.21.15.28:0/3323288652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a840051c0 msgr2=0x7f1a8400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.922+0000 7f1aa46d9640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a840051c0 0x7f1a8400b9c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c009fc0 tx=0x7f1a84005830 comp rx=0 tx=0).stop 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.922+0000 7f1aa46d9640 1 -- 172.21.15.28:0/3323288652 shutdown_connections 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.922+0000 7f1aa46d9640 1 --2- 172.21.15.28:0/3323288652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a840051c0 0x7f1a8400b9c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.922+0000 7f1aa46d9640 1 -- 172.21.15.28:0/3323288652 >> 172.21.15.28:0/3323288652 conn(0x7f1a9c0d2980 msgr2=0x7f1a9c0d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.922+0000 7f1aa46d9640 1 -- 172.21.15.28:0/3323288652 shutdown_connections 2024-04-25T19:12:29.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.922+0000 7f1aa46d9640 1 -- 172.21.15.28:0/3323288652 wait complete. 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.924+0000 7f1aa46d9640 1 Processor -- start 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.924+0000 7f1aa46d9640 1 -- start start 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa46d9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a840051c0 0x7f1a9c150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa46d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c0cb540 0x7f1a9c0cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa46d9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a9c15e8f0 0x7f1a9c15ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa46d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c0b53c0 con 0x7f1a9c0cb540 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa46d9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9c0b5240 con 0x7f1a9c15e8f0 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa46d9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9c164800 con 0x7f1a840051c0 2024-04-25T19:12:29.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa1c74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c0cb540 0x7f1a9c0cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa1c74640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c0cb540 0x7f1a9c0cb950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38498/0 (socket says 172.21.15.28:38498) 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa1c74640 1 -- 172.21.15.28:0/2855936600 learned_addr learned my addr 172.21.15.28:0/2855936600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa2c76640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a9c15e8f0 0x7f1a9c15ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa1c74640 1 -- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a840051c0 msgr2=0x7f1a9c150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa2475640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a840051c0 0x7f1a9c150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa1c74640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a840051c0 0x7f1a9c150fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa1c74640 1 -- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a9c15e8f0 msgr2=0x7f1a9c15ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa1c74640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a9c15e8f0 0x7f1a9c15ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa1c74640 1 -- 172.21.15.28:0/2855936600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9c15f240 con 0x7f1a9c0cb540 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.925+0000 7f1aa2c76640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a9c15e8f0 0x7f1a9c15ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.926+0000 7f1aa1c74640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c0cb540 0x7f1a9c0cb950 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000f5c0 tx=0x7f1a9000fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.926+0000 7f1a9affd640 1 -- 172.21.15.28:0/2855936600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9001a070 con 0x7f1a9c0cb540 2024-04-25T19:12:29.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.926+0000 7f1a9affd640 1 -- 172.21.15.28:0/2855936600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a90005320 con 0x7f1a9c0cb540 2024-04-25T19:12:29.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.926+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a9c0b9130 con 0x7f1a9c0cb540 2024-04-25T19:12:29.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.926+0000 7f1a9affd640 1 -- 172.21.15.28:0/2855936600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a90006a40 con 0x7f1a9c0cb540 2024-04-25T19:12:29.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.926+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a9c0b96f0 con 0x7f1a9c0cb540 2024-04-25T19:12:29.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.927+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a9c050740 con 0x7f1a9c0cb540 2024-04-25T19:12:29.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.927+0000 7f1a9affd640 1 -- 172.21.15.28:0/2855936600 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1a9001d070 con 0x7f1a9c0cb540 2024-04-25T19:12:29.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.928+0000 7f1a9affd640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1a880744b0 0x7f1a880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.928+0000 7f1a9affd640 1 -- 172.21.15.28:0/2855936600 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f1a90094e00 con 0x7f1a9c0cb540 2024-04-25T19:12:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.928+0000 7f1aa2475640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1a880744b0 0x7f1a880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:29.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.928+0000 7f1aa2475640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1a880744b0 0x7f1a880769c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a94001050 tx=0x7f1a94009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:29.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:29.933+0000 7f1a9affd640 1 -- 172.21.15.28:0/2855936600 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a9c15f240 con 0x7f1a9c0cb540 2024-04-25T19:12:30.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.119+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1a9c0cc000 con 0x7f1a9c0cb540 2024-04-25T19:12:30.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.120+0000 7f1a9affd640 1 -- 172.21.15.28:0/2855936600 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f1a9005f190 con 0x7f1a9c0cb540 2024-04-25T19:12:30.122 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1a880744b0 msgr2=0x7f1a880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1a880744b0 0x7f1a880769c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a94001050 tx=0x7f1a94009040 comp rx=0 tx=0).stop 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c0cb540 msgr2=0x7f1a9c0cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c0cb540 0x7f1a9c0cb950 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1a9000f5c0 tx=0x7f1a9000fa80 comp rx=0 tx=0).stop 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 shutdown_connections 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f1a880744b0 0x7f1a880769c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a9c15e8f0 0x7f1a9c15ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a9c0cb540 0x7f1a9c0cb950 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 --2- 172.21.15.28:0/2855936600 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f1a840051c0 0x7f1a9c150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 >> 172.21.15.28:0/2855936600 conn(0x7f1a9c0d2980 msgr2=0x7f1a9c0b6150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:30.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 shutdown_connections 2024-04-25T19:12:30.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.122+0000 7f1aa46d9640 1 -- 172.21.15.28:0/2855936600 wait complete. 2024-04-25T19:12:30.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T19:12:30.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.247+0000 7f0320e51640 1 Processor -- start 2024-04-25T19:12:30.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.247+0000 7f0320e51640 1 -- start start 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f0320e51640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f031c0d0ee0 0x7f031c0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f0320e51640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f031c06f5b0 con 0x7f031c15aaa0 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f0320e51640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f031c0d14a0 con 0x7f031c0b5020 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f0320e51640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f031c0af290 con 0x7f031c0d0ee0 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f031a59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f031c15aaa0 0x7f031c0b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36976/0 (socket says 172.21.15.28:36976) 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f031a59c640 1 -- 172.21.15.28:0/3798125719 learned_addr learned my addr 172.21.15.28:0/3798125719 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f0319d9b640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f031c0d0ee0 0x7f031c0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3399691331 0 0) 0x7f031c06f5b0 con 0x7f031c15aaa0 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.249+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0308003520 con 0x7f031c15aaa0 2024-04-25T19:12:30.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1191967485 0 0) 0x7f0308003520 con 0x7f031c15aaa0 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f031c0d0ee0 msgr2=0x7f031c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f031c0d0ee0 0x7f031c0affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 >> v1:172.21.15.53:6789/0 conn(0x7f031c0b5020 legacy=0x7f031c160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031c0af4e0 con 0x7f031c15aaa0 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f0314001670 con 0x7f031c15aaa0 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 >> v1:172.21.15.28:6789/0 conn(0x7f031c15aaa0 legacy=0x7f031c0b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03080039e0 0x7f0308003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f031c0d0ee0 0x7f0308004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0308004d40 0x7f0308005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f031c0af290 con 0x7f0308004d40 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f030800baa0 con 0x7f03080039e0 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f030800deb0 con 0x7f031c0d0ee0 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031ad9d640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0308004d40 0x7f0308005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031a59c640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03080039e0 0x7f0308003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f0319d9b640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f031c0d0ee0 0x7f0308004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031ad9d640 1 -- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f031c0d0ee0 msgr2=0x7f0308004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031ad9d640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f031c0d0ee0 0x7f0308004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031ad9d640 1 -- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03080039e0 msgr2=0x7f0308003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031ad9d640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03080039e0 0x7f0308003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031ad9d640 1 -- 172.21.15.28:0/3798125719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0308003520 con 0x7f0308004d40 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.250+0000 7f031a59c640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f03080039e0 0x7f0308003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.251+0000 7f031ad9d640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0308004d40 0x7f0308005150 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f031c06ebc0 tx=0x7f0308005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2732c3674eaadc47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.251+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c037070 con 0x7f0308004d40 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.251+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030c032440 con 0x7f0308004d40 2024-04-25T19:12:30.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.251+0000 7f031959a640 1 -- 172.21.15.28:0/3798125719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c001b30 con 0x7f0308004d40 2024-04-25T19:12:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.252+0000 7f0320e51640 1 -- 172.21.15.28:0/3798125719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0308004d40 msgr2=0x7f0308005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.252+0000 7f0320e51640 1 --2- 172.21.15.28:0/3798125719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0308004d40 0x7f0308005150 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f031c06ebc0 tx=0x7f0308005830 comp rx=0 tx=0).stop 2024-04-25T19:12:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.252+0000 7f031ad9d640 1 -- 172.21.15.28:0/3798125719 reap_dead start 2024-04-25T19:12:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.252+0000 7f0320e51640 1 -- 172.21.15.28:0/3798125719 shutdown_connections 2024-04-25T19:12:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.252+0000 7f0320e51640 1 -- 172.21.15.28:0/3798125719 >> 172.21.15.28:0/3798125719 conn(0x7f031c0742c0 msgr2=0x7f031c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.252+0000 7f0320e51640 1 -- 172.21.15.28:0/3798125719 shutdown_connections 2024-04-25T19:12:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.252+0000 7f0320e51640 1 -- 172.21.15.28:0/3798125719 wait complete. 2024-04-25T19:12:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.254+0000 7f0320e51640 1 Processor -- start 2024-04-25T19:12:30.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.254+0000 7f0320e51640 1 -- start start 2024-04-25T19:12:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0320e51640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0308004d40 0x7f031c0b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0320e51640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031c0b2da0 0x7f031c1f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0320e51640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f031c1f5850 0x7f031c153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0320e51640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f031c0b5430 con 0x7f031c0b2da0 2024-04-25T19:12:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0320e51640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f031c0afaa0 con 0x7f031c1f5850 2024-04-25T19:12:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0320e51640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f031c0b55b0 con 0x7f0308004d40 2024-04-25T19:12:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031c0b2da0 0x7f031c1f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0319d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031c0b2da0 0x7f031c1f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38516/0 (socket says 172.21.15.28:38516) 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f0319d9b640 1 -- 172.21.15.28:0/3430378911 learned_addr learned my addr 172.21.15.28:0/3430378911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f031a59c640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0308004d40 0x7f031c0b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.255+0000 7f031ad9d640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f031c1f5850 0x7f031c153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0319d9b640 1 -- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0308004d40 msgr2=0x7f031c0b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0319d9b640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0308004d40 0x7f031c0b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0319d9b640 1 -- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f031c1f5850 msgr2=0x7f031c153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0319d9b640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f031c1f5850 0x7f031c153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0319d9b640 1 -- 172.21.15.28:0/3430378911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031c1f6220 con 0x7f031c0b2da0 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f031ad9d640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f031c1f5850 0x7f031c153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f031a59c640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0308004d40 0x7f031c0b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0319d9b640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031c0b2da0 0x7f031c1f5310 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0304011940 tx=0x7f0304011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0312ffd640 1 -- 172.21.15.28:0/3430378911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030400faf0 con 0x7f031c0b2da0 2024-04-25T19:12:30.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0312ffd640 1 -- 172.21.15.28:0/3430378911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030400fc90 con 0x7f031c0b2da0 2024-04-25T19:12:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f031c1f5f40 con 0x7f031c0b2da0 2024-04-25T19:12:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0312ffd640 1 -- 172.21.15.28:0/3430378911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0304014440 con 0x7f031c0b2da0 2024-04-25T19:12:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.256+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f031c146d20 con 0x7f031c0b2da0 2024-04-25T19:12:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.257+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f031c1f5cb0 con 0x7f031c0b2da0 2024-04-25T19:12:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.258+0000 7f0312ffd640 1 -- 172.21.15.28:0/3430378911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0304012040 con 0x7f031c0b2da0 2024-04-25T19:12:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.258+0000 7f0312ffd640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f02fc0744b0 0x7f02fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.258+0000 7f0312ffd640 1 -- 172.21.15.28:0/3430378911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0304095bf0 con 0x7f031c0b2da0 2024-04-25T19:12:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.258+0000 7f031a59c640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f02fc0744b0 0x7f02fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.259+0000 7f031a59c640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f02fc0744b0 0x7f02fc0769c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0314001710 tx=0x7f0314009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.263+0000 7f0312ffd640 1 -- 172.21.15.28:0/3430378911 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0304019030 con 0x7f031c0b2da0 2024-04-25T19:12:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.452+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f031c050740 con 0x7f031c0b2da0 2024-04-25T19:12:30.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.452+0000 7f0312ffd640 1 -- 172.21.15.28:0/3430378911 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f03040600a0 con 0x7f031c0b2da0 2024-04-25T19:12:30.454 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f02fc0744b0 msgr2=0x7f02fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f02fc0744b0 0x7f02fc0769c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0314001710 tx=0x7f0314009040 comp rx=0 tx=0).stop 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031c0b2da0 msgr2=0x7f031c1f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031c0b2da0 0x7f031c1f5310 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0304011940 tx=0x7f0304011e00 comp rx=0 tx=0).stop 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 shutdown_connections 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f02fc0744b0 0x7f02fc0769c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f031c1f5850 0x7f031c153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031c0b2da0 0x7f031c1f5310 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 --2- 172.21.15.28:0/3430378911 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0308004d40 0x7f031c0b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 >> 172.21.15.28:0/3430378911 conn(0x7f031c0742c0 msgr2=0x7f031c156cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 shutdown_connections 2024-04-25T19:12:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.455+0000 7f0320e51640 1 -- 172.21.15.28:0/3430378911 wait complete. 2024-04-25T19:12:30.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.577+0000 7fcff8d84640 1 Processor -- start 2024-04-25T19:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.577+0000 7fcff8d84640 1 -- start start 2024-04-25T19:12:30.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff8d84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff40cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff8d84640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcff40cef50 0x7fcff40d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff8d84640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff406d710 con 0x7fcff409d010 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff8d84640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fcff408cc20 con 0x7fcff40cef50 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff8d84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcff408ce70 con 0x7fcff409cb60 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff259c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcff409d010 0x7fcff406ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36982/0 (socket says 172.21.15.28:36982) 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff259c640 1 -- 172.21.15.28:0/2334141893 learned_addr learned my addr 172.21.15.28:0/2334141893 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff2d9d640 1 --2- 172.21.15.28:0/2334141893 >> v2:172.21.15.28:3300/0 conn(0x7fcff40cef50 0x7fcff40d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff1d9b640 1 --2- 172.21.15.28:0/2334141893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff40cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff2d9d640 1 -- 172.21.15.28:0/2334141893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 msgr2=0x7fcff40cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff2d9d640 1 --2- 172.21.15.28:0/2334141893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff40cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff2d9d640 1 -- 172.21.15.28:0/2334141893 >> v1:172.21.15.28:6789/0 conn(0x7fcff409d010 legacy=0x7fcff406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff2d9d640 1 -- 172.21.15.28:0/2334141893 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff40b92d0 con 0x7fcff40cef50 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff159a640 1 -- 172.21.15.28:0/2334141893 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1274978506 0 0) 0x7fcff406d710 con 0x7fcff409d010 2024-04-25T19:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.579+0000 7fcff1d9b640 1 --2- 172.21.15.28:0/2334141893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff40cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.580+0000 7fcff2d9d640 1 --2- 172.21.15.28:0/2334141893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff40cef50 0x7fcff40d1cb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009d90 tx=0x7fcfe402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6fa30e5cb5a4b4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.580+0000 7fcff159a640 1 -- 172.21.15.28:0/2334141893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4007740 con 0x7fcff40cef50 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.580+0000 7fcff159a640 1 -- 172.21.15.28:0/2334141893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfe40078e0 con 0x7fcff40cef50 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.580+0000 7fcff159a640 1 -- 172.21.15.28:0/2334141893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4007cb0 con 0x7fcff40cef50 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.581+0000 7fcff8d84640 1 -- 172.21.15.28:0/2334141893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff40cef50 msgr2=0x7fcff40d1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.581+0000 7fcff8d84640 1 --2- 172.21.15.28:0/2334141893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff40cef50 0x7fcff40d1cb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009d90 tx=0x7fcfe402a660 comp rx=0 tx=0).stop 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.581+0000 7fcff8d84640 1 -- 172.21.15.28:0/2334141893 shutdown_connections 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.581+0000 7fcff8d84640 1 --2- 172.21.15.28:0/2334141893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff40cef50 0x7fcff40d1cb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.581+0000 7fcff8d84640 1 --2- 172.21.15.28:0/2334141893 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff40cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.581+0000 7fcff8d84640 1 -- 172.21.15.28:0/2334141893 >> 172.21.15.28:0/2334141893 conn(0x7fcff40d0430 msgr2=0x7fcff40d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.581+0000 7fcff8d84640 1 -- 172.21.15.28:0/2334141893 shutdown_connections 2024-04-25T19:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.581+0000 7fcff8d84640 1 -- 172.21.15.28:0/2334141893 wait complete. 2024-04-25T19:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.583+0000 7fcff8d84640 1 Processor -- start 2024-04-25T19:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.583+0000 7fcff8d84640 1 -- start start 2024-04-25T19:12:30.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff8d84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff41dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff8d84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff409d010 0x7fcff409f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff8d84640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcff40cef50 0x7fcff409fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff8d84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcff406ea20 con 0x7fcff409d010 2024-04-25T19:12:30.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff8d84640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcff406e5e0 con 0x7fcff40cef50 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff8d84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcff406eba0 con 0x7fcff409cb60 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff409d010 0x7fcff409f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff409d010 0x7fcff409f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38540/0 (socket says 172.21.15.28:38540) 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.584+0000 7fcff1d9b640 1 -- 172.21.15.28:0/3442158084 learned_addr learned my addr 172.21.15.28:0/3442158084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff259c640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff41dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff2d9d640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcff40cef50 0x7fcff409fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff1d9b640 1 -- 172.21.15.28:0/3442158084 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 msgr2=0x7fcff41dbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff1d9b640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff41dbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff1d9b640 1 -- 172.21.15.28:0/3442158084 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcff40cef50 msgr2=0x7fcff409fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff1d9b640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcff40cef50 0x7fcff409fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff1d9b640 1 -- 172.21.15.28:0/3442158084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff40b1390 con 0x7fcff409d010 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff1d9b640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff409d010 0x7fcff409f930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00d800 tx=0x7fcfdc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcfeaffd640 1 -- 172.21.15.28:0/3442158084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc004030 con 0x7fcff409d010 2024-04-25T19:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcfeaffd640 1 -- 172.21.15.28:0/3442158084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfdc0041d0 con 0x7fcff409d010 2024-04-25T19:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff40b1670 con 0x7fcff409d010 2024-04-25T19:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcfeaffd640 1 -- 172.21.15.28:0/3442158084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc0096a0 con 0x7fcff409d010 2024-04-25T19:12:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.585+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff40b1c30 con 0x7fcff409d010 2024-04-25T19:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.586+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff40cfc70 con 0x7fcff409d010 2024-04-25T19:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.587+0000 7fcfeaffd640 1 -- 172.21.15.28:0/3442158084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcfdc009840 con 0x7fcff409d010 2024-04-25T19:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.587+0000 7fcfeaffd640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcfd40744b0 0x7fcfd40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.588+0000 7fcfeaffd640 1 -- 172.21.15.28:0/3442158084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcfdc095950 con 0x7fcff409d010 2024-04-25T19:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.588+0000 7fcff259c640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcfd40744b0 0x7fcfd40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.588+0000 7fcff259c640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcfd40744b0 0x7fcfd40769c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcfec001000 tx=0x7fcfec009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.592+0000 7fcfeaffd640 1 -- 172.21.15.28:0/3442158084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcff40cfc70 con 0x7fcff409d010 2024-04-25T19:12:30.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.815+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcff40b1100 con 0x7fcff409d010 2024-04-25T19:12:30.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.815+0000 7fcfeaffd640 1 -- 172.21.15.28:0/3442158084 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7fcfdc05fc40 con 0x7fcff409d010 2024-04-25T19:12:30.817 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:30.818 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:51.367677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:12:30.818 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T19:12:30.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcfd40744b0 msgr2=0x7fcfd40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcfd40744b0 0x7fcfd40769c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fcfec001000 tx=0x7fcfec009040 comp rx=0 tx=0).stop 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff409d010 msgr2=0x7fcff409f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff409d010 0x7fcff409f930 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00d800 tx=0x7fcfdc00dcc0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 shutdown_connections 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcfd40744b0 0x7fcfd40769c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcff40cef50 0x7fcff409fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcff409d010 0x7fcff409f930 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 --2- 172.21.15.28:0/3442158084 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcff409cb60 0x7fcff41dbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.817+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 >> 172.21.15.28:0/3442158084 conn(0x7fcff40d0430 msgr2=0x7fcff40b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.818+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 shutdown_connections 2024-04-25T19:12:30.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.818+0000 7fcff8d84640 1 -- 172.21.15.28:0/3442158084 wait complete. 2024-04-25T19:12:30.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.944+0000 7fcaf659d640 1 Processor -- start 2024-04-25T19:12:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.944+0000 7fcaf659d640 1 -- start start 2024-04-25T19:12:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.945+0000 7fcaf659d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcaf006d580 0x7fcaf006d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.945+0000 7fcaf659d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006f390 0x7fcaf006f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.945+0000 7fcaf659d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaf00b55a0 con 0x7fcaf00b5190 2024-04-25T19:12:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.945+0000 7fcaf659d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fcaf006ded0 con 0x7fcaf006d580 2024-04-25T19:12:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.945+0000 7fcaf659d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf015d560 con 0x7fcaf006f390 2024-04-25T19:12:30.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaf4b3a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006f390 0x7fcaf006f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaef7fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcaf006d580 0x7fcaf006d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaeffff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fcaf00b5190 0x7fcaf00d0ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:37364/0 (socket says 172.21.15.28:37364) 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaf4b3a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006f390 0x7fcaf006f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:44622/0 (socket says 172.21.15.28:44622) 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaeffff640 1 -- 172.21.15.28:0/1016351116 learned_addr learned my addr 172.21.15.28:0/1016351116 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1016351116 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 206728851 0 0) 0x7fcaf00b55a0 con 0x7fcaf00b5190 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1016351116 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcadc003520 con 0x7fcaf00b5190 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaf4b3a640 1 -- 172.21.15.28:0/1016351116 >> v2:172.21.15.53:3300/0 conn(0x7fcaf006d580 msgr2=0x7fcaf006d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaf4b3a640 1 --2- 172.21.15.28:0/1016351116 >> v2:172.21.15.53:3300/0 conn(0x7fcaf006d580 0x7fcaf006d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaf4b3a640 1 -- 172.21.15.28:0/1016351116 >> v1:172.21.15.53:6789/0 conn(0x7fcaf00b5190 legacy=0x7fcaf00d0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaf4b3a640 1 -- 172.21.15.28:0/1016351116 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf015d820 con 0x7fcaf006f390 2024-04-25T19:12:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.946+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/1016351116 >> v2:172.21.15.53:3300/0 conn(0x7fcaf006d580 0x7fcaf006d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.947+0000 7fcaf4b3a640 1 --2- 172.21.15.28:0/1016351116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006f390 0x7fcaf006f7a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009d90 tx=0x7fcae802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a8042002adb6d9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.947+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1016351116 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8007740 con 0x7fcaf006f390 2024-04-25T19:12:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.947+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1016351116 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae80078e0 con 0x7fcaf006f390 2024-04-25T19:12:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.947+0000 7fcaeeffd640 1 -- 172.21.15.28:0/1016351116 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8007c90 con 0x7fcaf006f390 2024-04-25T19:12:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.948+0000 7fcaf659d640 1 -- 172.21.15.28:0/1016351116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006f390 msgr2=0x7fcaf006f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.948+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1016351116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006f390 0x7fcaf006f7a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009d90 tx=0x7fcae802a660 comp rx=0 tx=0).stop 2024-04-25T19:12:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.948+0000 7fcaf659d640 1 -- 172.21.15.28:0/1016351116 shutdown_connections 2024-04-25T19:12:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.948+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1016351116 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006f390 0x7fcaf006f7a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.948+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1016351116 >> v2:172.21.15.53:3300/0 conn(0x7fcaf006d580 0x7fcaf006d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.948+0000 7fcaf659d640 1 -- 172.21.15.28:0/1016351116 >> 172.21.15.28:0/1016351116 conn(0x7fcaf00732b0 msgr2=0x7fcaf00736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.948+0000 7fcaf659d640 1 -- 172.21.15.28:0/1016351116 shutdown_connections 2024-04-25T19:12:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.948+0000 7fcaf659d640 1 -- 172.21.15.28:0/1016351116 wait complete. 2024-04-25T19:12:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.950+0000 7fcaf659d640 1 Processor -- start 2024-04-25T19:12:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.950+0000 7fcaf659d640 1 -- start start 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf659d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006d580 0x7fcaf01f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf659d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf006f390 0x7fcaf014f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf659d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b5190 0x7fcaf00afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf659d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf015e0c0 con 0x7fcaf00b5190 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf659d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf015dde0 con 0x7fcaf006f390 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf659d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf009f8e0 con 0x7fcaf006d580 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf4b3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b5190 0x7fcaf00afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf4b3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b5190 0x7fcaf00afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49296/0 (socket says 172.21.15.28:49296) 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaf4b3a640 1 -- 172.21.15.28:0/1649231209 learned_addr learned my addr 172.21.15.28:0/1649231209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaef7fe640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf006f390 0x7fcaf014f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.951+0000 7fcaeffff640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006d580 0x7fcaf01f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaf4b3a640 1 -- 172.21.15.28:0/1649231209 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006d580 msgr2=0x7fcaf01f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaf4b3a640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006d580 0x7fcaf01f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaf4b3a640 1 -- 172.21.15.28:0/1649231209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf006f390 msgr2=0x7fcaf014f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaf4b3a640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf006f390 0x7fcaf014f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaf4b3a640 1 -- 172.21.15.28:0/1649231209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf014fb50 con 0x7fcaf00b5190 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaf4b3a640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b5190 0x7fcaf00afce0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009eb0 tx=0x7fcae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaecff9640 1 -- 172.21.15.28:0/1649231209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae803f070 con 0x7fcaf00b5190 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaecff9640 1 -- 172.21.15.28:0/1649231209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae8031c60 con 0x7fcaf00b5190 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaecff9640 1 -- 172.21.15.28:0/1649231209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8009450 con 0x7fcaf00b5190 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf01f2fb0 con 0x7fcaf00b5190 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.952+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf01f3570 con 0x7fcaf00b5190 2024-04-25T19:12:30.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.953+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf006d990 con 0x7fcaf00b5190 2024-04-25T19:12:30.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.954+0000 7fcaecff9640 1 -- 172.21.15.28:0/1649231209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcae803b410 con 0x7fcaf00b5190 2024-04-25T19:12:30.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.954+0000 7fcaecff9640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcad00744b0 0x7fcad00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:30.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.954+0000 7fcaecff9640 1 -- 172.21.15.28:0/1649231209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcae80b6530 con 0x7fcaf00b5190 2024-04-25T19:12:30.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.954+0000 7fcaeffff640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcad00744b0 0x7fcad00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:30.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.955+0000 7fcaeffff640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcad00744b0 0x7fcad00769c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcae00011e0 tx=0x7fcae0007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:30.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:30.959+0000 7fcaecff9640 1 -- 172.21.15.28:0/1649231209 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcaf006d990 con 0x7fcaf00b5190 2024-04-25T19:12:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.144+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcaf0071b60 con 0x7fcaf00b5190 2024-04-25T19:12:31.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.145+0000 7fcaecff9640 1 -- 172.21.15.28:0/1649231209 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7fcaf0071b60 con 0x7fcaf00b5190 2024-04-25T19:12:31.147 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:31.148 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcad00744b0 msgr2=0x7fcad00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcad00744b0 0x7fcad00769c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fcae00011e0 tx=0x7fcae0007d30 comp rx=0 tx=0).stop 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b5190 msgr2=0x7fcaf00afce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b5190 0x7fcaf00afce0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fcae8009eb0 tx=0x7fcae8004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 shutdown_connections 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fcad00744b0 0x7fcad00769c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcaf00b5190 0x7fcaf00afce0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcaf006f390 0x7fcaf014f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 --2- 172.21.15.28:0/1649231209 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fcaf006d580 0x7fcaf01f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 >> 172.21.15.28:0/1649231209 conn(0x7fcaf00732b0 msgr2=0x7fcaf0156ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 shutdown_connections 2024-04-25T19:12:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.148+0000 7fcaf659d640 1 -- 172.21.15.28:0/1649231209 wait complete. 2024-04-25T19:12:31.161 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T19:12:31.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:31.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.271+0000 7fee4bd97640 1 Processor -- start 2024-04-25T19:12:31.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.272+0000 7fee4bd97640 1 -- start start 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee4bd97640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fee4409caf0 0x7fee440d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee4bd97640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee4409cf00 0x7fee440ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee4bd97640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee440d1ce0 con 0x7fee440d2670 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee4bd97640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fee440b8fa0 con 0x7fee4409cf00 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee4bd97640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fee440cf8d0 con 0x7fee4409caf0 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee4a334640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee4409cf00 0x7fee440ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee4a334640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee4409cf00 0x7fee440ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49298/0 (socket says 172.21.15.28:49298) 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee4a334640 1 -- 172.21.15.28:0/355006309 learned_addr learned my addr 172.21.15.28:0/355006309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.273+0000 7fee49b33640 1 --2- 172.21.15.28:0/355006309 >> v2:172.21.15.53:3300/0 conn(0x7fee4409caf0 0x7fee440d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.274+0000 7fee4a334640 1 -- 172.21.15.28:0/355006309 >> v2:172.21.15.53:3300/0 conn(0x7fee4409caf0 msgr2=0x7fee440d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.274+0000 7fee4a334640 1 --2- 172.21.15.28:0/355006309 >> v2:172.21.15.53:3300/0 conn(0x7fee4409caf0 0x7fee440d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.274+0000 7fee4a334640 1 -- 172.21.15.28:0/355006309 >> v1:172.21.15.28:6789/0 conn(0x7fee440d2670 legacy=0x7fee440b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.274+0000 7fee4a334640 1 -- 172.21.15.28:0/355006309 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee440cfbb0 con 0x7fee4409cf00 2024-04-25T19:12:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.274+0000 7fee48b31640 1 -- 172.21.15.28:0/355006309 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1355674124 0 0) 0x7fee440d1ce0 con 0x7fee440d2670 2024-04-25T19:12:31.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.274+0000 7fee49b33640 1 --2- 172.21.15.28:0/355006309 >> v2:172.21.15.53:3300/0 conn(0x7fee4409caf0 0x7fee440d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.274+0000 7fee4a334640 1 --2- 172.21.15.28:0/355006309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4409cf00 0x7fee440ce720 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fee4408cd70 tx=0x7fee3402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c115d5d37cbe602f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.274+0000 7fee48b31640 1 -- 172.21.15.28:0/355006309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34036070 con 0x7fee4409cf00 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee48b31640 1 -- 172.21.15.28:0/355006309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee34031600 con 0x7fee4409cf00 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee48b31640 1 -- 172.21.15.28:0/355006309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee340319d0 con 0x7fee4409cf00 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee4bd97640 1 -- 172.21.15.28:0/355006309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4409cf00 msgr2=0x7fee440ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee4bd97640 1 --2- 172.21.15.28:0/355006309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4409cf00 0x7fee440ce720 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fee4408cd70 tx=0x7fee3402a6a0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee4bd97640 1 -- 172.21.15.28:0/355006309 shutdown_connections 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee4bd97640 1 --2- 172.21.15.28:0/355006309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee4409cf00 0x7fee440ce720 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee4bd97640 1 --2- 172.21.15.28:0/355006309 >> v2:172.21.15.53:3300/0 conn(0x7fee4409caf0 0x7fee440d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee4bd97640 1 -- 172.21.15.28:0/355006309 >> 172.21.15.28:0/355006309 conn(0x7fee440d03f0 msgr2=0x7fee440d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee4bd97640 1 -- 172.21.15.28:0/355006309 shutdown_connections 2024-04-25T19:12:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.275+0000 7fee4bd97640 1 -- 172.21.15.28:0/355006309 wait complete. 2024-04-25T19:12:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.277+0000 7fee4bd97640 1 Processor -- start 2024-04-25T19:12:31.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.277+0000 7fee4bd97640 1 -- start start 2024-04-25T19:12:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.278+0000 7fee4bd97640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee4409caf0 0x7fee440a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.278+0000 7fee4bd97640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4409cf00 0x7fee440a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.278+0000 7fee4bd97640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee440d2670 0x7fee441d5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.278+0000 7fee4bd97640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee440b9430 con 0x7fee440d2670 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.278+0000 7fee4bd97640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fee440b78e0 con 0x7fee4409cf00 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.278+0000 7fee4bd97640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fee440d4790 con 0x7fee4409caf0 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee440d2670 0x7fee441d5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee440d2670 0x7fee441d5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49312/0 (socket says 172.21.15.28:49312) 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 -- 172.21.15.28:0/1964389222 learned_addr learned my addr 172.21.15.28:0/1964389222 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee49332640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4409cf00 0x7fee440a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee49b33640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee4409caf0 0x7fee440a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 -- 172.21.15.28:0/1964389222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee4409caf0 msgr2=0x7fee440a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee4409caf0 0x7fee440a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 -- 172.21.15.28:0/1964389222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4409cf00 msgr2=0x7fee440a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4409cf00 0x7fee440a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 -- 172.21.15.28:0/1964389222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee440c8d60 con 0x7fee440d2670 2024-04-25T19:12:31.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4a334640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee440d2670 0x7fee441d5770 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fee34009f40 tx=0x7fee34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee427fc640 1 -- 172.21.15.28:0/1964389222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34036050 con 0x7fee440d2670 2024-04-25T19:12:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee427fc640 1 -- 172.21.15.28:0/1964389222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee34042070 con 0x7fee440d2670 2024-04-25T19:12:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee440c9040 con 0x7fee440d2670 2024-04-25T19:12:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee427fc640 1 -- 172.21.15.28:0/1964389222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee34009510 con 0x7fee440d2670 2024-04-25T19:12:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.279+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee440c9600 con 0x7fee440d2670 2024-04-25T19:12:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.280+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee44050740 con 0x7fee440d2670 2024-04-25T19:12:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.281+0000 7fee427fc640 1 -- 172.21.15.28:0/1964389222 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fee34007bf0 con 0x7fee440d2670 2024-04-25T19:12:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.282+0000 7fee427fc640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fee300744b0 0x7fee300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.282+0000 7fee49b33640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fee300744b0 0x7fee300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.282+0000 7fee427fc640 1 -- 172.21.15.28:0/1964389222 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fee340b6640 con 0x7fee440d2670 2024-04-25T19:12:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.282+0000 7fee49b33640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fee300744b0 0x7fee300769c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fee44069f40 tx=0x7fee3c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:31.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.286+0000 7fee427fc640 1 -- 172.21.15.28:0/1964389222 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee340809d0 con 0x7fee440d2670 2024-04-25T19:12:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.511+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee440b6a10 con 0x7fee440d2670 2024-04-25T19:12:31.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.512+0000 7fee427fc640 1 -- 172.21.15.28:0/1964389222 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7fee340802d0 con 0x7fee440d2670 2024-04-25T19:12:31.514 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:31.514 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:51.367677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:12:31.514 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T19:12:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fee300744b0 msgr2=0x7fee300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fee300744b0 0x7fee300769c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fee44069f40 tx=0x7fee3c008040 comp rx=0 tx=0).stop 2024-04-25T19:12:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee440d2670 msgr2=0x7fee441d5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee440d2670 0x7fee441d5770 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fee34009f40 tx=0x7fee34004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 shutdown_connections 2024-04-25T19:12:31.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fee300744b0 0x7fee300769c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee440d2670 0x7fee441d5770 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee4409cf00 0x7fee440a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 --2- 172.21.15.28:0/1964389222 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fee4409caf0 0x7fee440a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 >> 172.21.15.28:0/1964389222 conn(0x7fee440d03f0 msgr2=0x7fee44072160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 shutdown_connections 2024-04-25T19:12:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.514+0000 7fee4bd97640 1 -- 172.21.15.28:0/1964389222 wait complete. 2024-04-25T19:12:31.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:31.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.636+0000 7f64e0a61640 1 Processor -- start 2024-04-25T19:12:31.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.636+0000 7f64e0a61640 1 -- start start 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64e0a61640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc06e5e0 0x7f64dc161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64e0a61640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f64dc15f7d0 0x7f64dc15fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64e0a61640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64dc163e80 con 0x7f64dc0b5130 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64e0a61640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f64dc163c30 con 0x7f64dc15f7d0 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64e0a61640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f64dc164a90 con 0x7f64dc06e5e0 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64dad9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f64dc15f7d0 0x7f64dc15fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc06e5e0 0x7f64dc161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64da59c640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f64dc0b5130 0x7f64dc06ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:37368/0 (socket says 172.21.15.28:37368) 2024-04-25T19:12:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.638+0000 7f64da59c640 1 -- 172.21.15.28:0/3223360443 learned_addr learned my addr 172.21.15.28:0/3223360443 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.639+0000 7f64d9d9b640 1 -- 172.21.15.28:0/3223360443 >> v2:172.21.15.53:3300/0 conn(0x7f64dc15f7d0 msgr2=0x7f64dc15fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.639+0000 7f64d9d9b640 1 --2- 172.21.15.28:0/3223360443 >> v2:172.21.15.53:3300/0 conn(0x7f64dc15f7d0 0x7f64dc15fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.639+0000 7f64d9d9b640 1 -- 172.21.15.28:0/3223360443 >> v1:172.21.15.53:6789/0 conn(0x7f64dc0b5130 legacy=0x7f64dc06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.639+0000 7f64d9d9b640 1 -- 172.21.15.28:0/3223360443 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc164160 con 0x7f64dc06e5e0 2024-04-25T19:12:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.639+0000 7f64dad9d640 1 --2- 172.21.15.28:0/3223360443 >> v2:172.21.15.53:3300/0 conn(0x7f64dc15f7d0 0x7f64dc15fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.639+0000 7f64d9d9b640 1 --2- 172.21.15.28:0/3223360443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc06e5e0 0x7f64dc161130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097a0 tx=0x7f64c402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97884fa28e299373 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.639+0000 7f64d959a640 1 -- 172.21.15.28:0/3223360443 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4037070 con 0x7f64dc06e5e0 2024-04-25T19:12:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64d959a640 1 -- 172.21.15.28:0/3223360443 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c4032440 con 0x7f64dc06e5e0 2024-04-25T19:12:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64d959a640 1 -- 172.21.15.28:0/3223360443 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4032810 con 0x7f64dc06e5e0 2024-04-25T19:12:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64e0a61640 1 -- 172.21.15.28:0/3223360443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc06e5e0 msgr2=0x7f64dc161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64e0a61640 1 --2- 172.21.15.28:0/3223360443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc06e5e0 0x7f64dc161130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097a0 tx=0x7f64c402a180 comp rx=0 tx=0).stop 2024-04-25T19:12:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64e0a61640 1 -- 172.21.15.28:0/3223360443 shutdown_connections 2024-04-25T19:12:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64e0a61640 1 --2- 172.21.15.28:0/3223360443 >> v2:172.21.15.53:3300/0 conn(0x7f64dc15f7d0 0x7f64dc15fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64e0a61640 1 --2- 172.21.15.28:0/3223360443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc06e5e0 0x7f64dc161130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64e0a61640 1 -- 172.21.15.28:0/3223360443 >> 172.21.15.28:0/3223360443 conn(0x7f64dc0b7ad0 msgr2=0x7f64dc0b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64e0a61640 1 -- 172.21.15.28:0/3223360443 shutdown_connections 2024-04-25T19:12:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.640+0000 7f64e0a61640 1 -- 172.21.15.28:0/3223360443 wait complete. 2024-04-25T19:12:31.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.642+0000 7f64e0a61640 1 Processor -- start 2024-04-25T19:12:31.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.642+0000 7f64e0a61640 1 -- start start 2024-04-25T19:12:31.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.643+0000 7f64e0a61640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64dc0b5130 0x7f64dc1555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.643+0000 7f64e0a61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64dc15f7d0 0x7f64dc155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.643+0000 7f64e0a61640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc157fc0 0x7f64dc0c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.643+0000 7f64e0a61640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc160120 con 0x7f64dc15f7d0 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.643+0000 7f64e0a61640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f64dc164720 con 0x7f64dc0b5130 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.643+0000 7f64e0a61640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f64dc1602a0 con 0x7f64dc157fc0 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64dc15f7d0 0x7f64dc155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64dc15f7d0 0x7f64dc155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49314/0 (socket says 172.21.15.28:49314) 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 -- 172.21.15.28:0/1658032383 learned_addr learned my addr 172.21.15.28:0/1658032383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64dad9d640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc157fc0 0x7f64dc0c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64da59c640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64dc0b5130 0x7f64dc1555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 -- 172.21.15.28:0/1658032383 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc157fc0 msgr2=0x7f64dc0c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc157fc0 0x7f64dc0c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 -- 172.21.15.28:0/1658032383 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64dc0b5130 msgr2=0x7f64dc1555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64dc0b5130 0x7f64dc1555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 -- 172.21.15.28:0/1658032383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64dc156080 con 0x7f64dc15f7d0 2024-04-25T19:12:31.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d9d9b640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64dc15f7d0 0x7f64dc155b40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f64c4035040 tx=0x7f64c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:31.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d2ffd640 1 -- 172.21.15.28:0/1658032383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4037040 con 0x7f64dc15f7d0 2024-04-25T19:12:31.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d2ffd640 1 -- 172.21.15.28:0/1658032383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c4030500 con 0x7f64dc15f7d0 2024-04-25T19:12:31.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64dc15ae70 con 0x7f64dc15f7d0 2024-04-25T19:12:31.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.644+0000 7f64d2ffd640 1 -- 172.21.15.28:0/1658032383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c40308d0 con 0x7f64dc15f7d0 2024-04-25T19:12:31.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.645+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64dc156290 con 0x7f64dc15f7d0 2024-04-25T19:12:31.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.645+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64dc158510 con 0x7f64dc15f7d0 2024-04-25T19:12:31.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.646+0000 7f64d2ffd640 1 -- 172.21.15.28:0/1658032383 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f64c4044020 con 0x7f64dc15f7d0 2024-04-25T19:12:31.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.647+0000 7f64d2ffd640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f64bc0744b0 0x7f64bc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:31.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.647+0000 7f64d2ffd640 1 -- 172.21.15.28:0/1658032383 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f64c40b5840 con 0x7f64dc15f7d0 2024-04-25T19:12:31.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.647+0000 7f64da59c640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f64bc0744b0 0x7f64bc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:31.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.647+0000 7f64da59c640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f64bc0744b0 0x7f64bc0769c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f64d4007fd0 tx=0x7f64d4007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.651+0000 7f64d2ffd640 1 -- 172.21.15.28:0/1658032383 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f64c407f570 con 0x7f64dc15f7d0 2024-04-25T19:12:31.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.877+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64dc050740 con 0x7f64dc15f7d0 2024-04-25T19:12:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.878+0000 7f64d2ffd640 1 -- 172.21.15.28:0/1658032383 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5174 (secure 0 0 0) 0x7f64c4083870 con 0x7f64dc15f7d0 2024-04-25T19:12:31.880 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:31.880 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/3883412989","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3883412989},{"type":"v1","addr":"172.21.15.53:6833","nonce":3883412989}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/336187649","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":336187649},{"type":"v1","addr":"172.21.15.28:6836","nonce":336187649}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/2501832860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":2501832860},{"type":"v1","addr":"172.21.15.28:6837","nonce":2501832860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:11:29.309189+0000","modified":"2024-04-25T19:11:51.367677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4281},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4281":{"gid":4281,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/3585928833","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3585928833},{"type":"v1","addr":"172.21.15.53:6835","nonce":3585928833}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4281,"qdb_cluster":[4281]},"id":1}]} 2024-04-25T19:12:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-25T19:12:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f64bc0744b0 msgr2=0x7f64bc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f64bc0744b0 0x7f64bc0769c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f64d4007fd0 tx=0x7f64d4007e80 comp rx=0 tx=0).stop 2024-04-25T19:12:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64dc15f7d0 msgr2=0x7f64dc155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64dc15f7d0 0x7f64dc155b40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f64c4035040 tx=0x7f64c40046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 shutdown_connections 2024-04-25T19:12:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f64bc0744b0 0x7f64bc0769c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f64dc157fc0 0x7f64dc0c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64dc15f7d0 0x7f64dc155b40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 --2- 172.21.15.28:0/1658032383 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64dc0b5130 0x7f64dc1555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:31.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 >> 172.21.15.28:0/1658032383 conn(0x7f64dc0b7ad0 msgr2=0x7f64dc0b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:31.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 shutdown_connections 2024-04-25T19:12:31.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:31.880+0000 7f64e0a61640 1 -- 172.21.15.28:0/1658032383 wait complete. 2024-04-25T19:12:31.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:32.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.003+0000 7f317c096640 1 Processor -- start 2024-04-25T19:12:32.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.003+0000 7f317c096640 1 -- start start 2024-04-25T19:12:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317c096640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f317406e5e0 0x7f3174161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317c096640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f317415f7d0 0x7f317415fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317c096640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3174163e80 con 0x7f31740b5130 2024-04-25T19:12:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317c096640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f3174163c30 con 0x7f317415f7d0 2024-04-25T19:12:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317c096640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f3174164a90 con 0x7f317406e5e0 2024-04-25T19:12:32.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317a633640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f317415f7d0 0x7f317415fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317a633640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f317415f7d0 0x7f317415fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49330/0 (socket says 172.21.15.28:49330) 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f3179e32640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f31740b5130 0x7f317406ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36858/0 (socket says 172.21.15.28:36858) 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317a633640 1 -- 172.21.15.28:0/3485657450 learned_addr learned my addr 172.21.15.28:0/3485657450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f3179631640 1 --2- 172.21.15.28:0/3485657450 >> v2:172.21.15.53:3300/0 conn(0x7f317406e5e0 0x7f3174161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317a633640 1 -- 172.21.15.28:0/3485657450 >> v2:172.21.15.53:3300/0 conn(0x7f317406e5e0 msgr2=0x7f3174161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317a633640 1 --2- 172.21.15.28:0/3485657450 >> v2:172.21.15.53:3300/0 conn(0x7f317406e5e0 0x7f3174161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317a633640 1 -- 172.21.15.28:0/3485657450 >> v1:172.21.15.28:6789/0 conn(0x7f31740b5130 legacy=0x7f317406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f3178e30640 1 -- 172.21.15.28:0/3485657450 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3343105903 0 0) 0x7f3174163e80 con 0x7f31740b5130 2024-04-25T19:12:32.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.005+0000 7f317a633640 1 -- 172.21.15.28:0/3485657450 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3174164160 con 0x7f317415f7d0 2024-04-25T19:12:32.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.006+0000 7f317a633640 1 --2- 172.21.15.28:0/3485657450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f317415fbe0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f317408e180 tx=0x7f316402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d246393eb79b3164 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.006+0000 7f3178e30640 1 -- 172.21.15.28:0/3485657450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3164036070 con 0x7f317415f7d0 2024-04-25T19:12:32.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.006+0000 7f3178e30640 1 -- 172.21.15.28:0/3485657450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3164031600 con 0x7f317415f7d0 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.006+0000 7f3178e30640 1 -- 172.21.15.28:0/3485657450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31640319d0 con 0x7f317415f7d0 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.007+0000 7f317c096640 1 -- 172.21.15.28:0/3485657450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 msgr2=0x7f317415fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.007+0000 7f317c096640 1 --2- 172.21.15.28:0/3485657450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f317415fbe0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f317408e180 tx=0x7f316402a730 comp rx=0 tx=0).stop 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.007+0000 7f317c096640 1 -- 172.21.15.28:0/3485657450 shutdown_connections 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.007+0000 7f317c096640 1 --2- 172.21.15.28:0/3485657450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f317415fbe0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.007+0000 7f317c096640 1 --2- 172.21.15.28:0/3485657450 >> v2:172.21.15.53:3300/0 conn(0x7f317406e5e0 0x7f3174161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.007+0000 7f317c096640 1 -- 172.21.15.28:0/3485657450 >> 172.21.15.28:0/3485657450 conn(0x7f31740b7ad0 msgr2=0x7f31740b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.007+0000 7f317c096640 1 -- 172.21.15.28:0/3485657450 shutdown_connections 2024-04-25T19:12:32.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.007+0000 7f317c096640 1 -- 172.21.15.28:0/3485657450 wait complete. 2024-04-25T19:12:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.009+0000 7f317c096640 1 Processor -- start 2024-04-25T19:12:32.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.009+0000 7f317c096640 1 -- start start 2024-04-25T19:12:32.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f317c096640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31740b5130 0x7f317415b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f317c096640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f31740ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f317c096640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31741f65c0 0x7f31740b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f317c096640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3174160120 con 0x7f317415f7d0 2024-04-25T19:12:32.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f317c096640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3174164720 con 0x7f31741f65c0 2024-04-25T19:12:32.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f317c096640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f31741602a0 con 0x7f31740b5130 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f3179631640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f31740ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f3179631640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f31740ae270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49338/0 (socket says 172.21.15.28:49338) 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f3179631640 1 -- 172.21.15.28:0/27304615 learned_addr learned my addr 172.21.15.28:0/27304615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f317a633640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31741f65c0 0x7f31740b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f3179e32640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31740b5130 0x7f317415b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f3179631640 1 -- 172.21.15.28:0/27304615 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31740b5130 msgr2=0x7f317415b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f3179631640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31740b5130 0x7f317415b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f3179631640 1 -- 172.21.15.28:0/27304615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31741f65c0 msgr2=0x7f31740b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.010+0000 7f3179631640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31741f65c0 0x7f31740b34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.011+0000 7f3179631640 1 -- 172.21.15.28:0/27304615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31741f70f0 con 0x7f317415f7d0 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.011+0000 7f3179631640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f31740ae270 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f316800e870 tx=0x7f316800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.011+0000 7f31727fc640 1 -- 172.21.15.28:0/27304615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168017070 con 0x7f317415f7d0 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.011+0000 7f31727fc640 1 -- 172.21.15.28:0/27304615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f316800c940 con 0x7f317415f7d0 2024-04-25T19:12:32.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.011+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31741f73d0 con 0x7f317415f7d0 2024-04-25T19:12:32.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.011+0000 7f31727fc640 1 -- 172.21.15.28:0/27304615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f316800cd30 con 0x7f317415f7d0 2024-04-25T19:12:32.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.011+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f317415b9e0 con 0x7f317415f7d0 2024-04-25T19:12:32.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.012+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3174050740 con 0x7f317415f7d0 2024-04-25T19:12:32.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.013+0000 7f31727fc640 1 -- 172.21.15.28:0/27304615 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f316801a070 con 0x7f317415f7d0 2024-04-25T19:12:32.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.013+0000 7f31727fc640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31600744b0 0x7f31600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.013+0000 7f31727fc640 1 -- 172.21.15.28:0/27304615 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3168095790 con 0x7f317415f7d0 2024-04-25T19:12:32.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.013+0000 7f3179e32640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31600744b0 0x7f31600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.014+0000 7f3179e32640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31600744b0 0x7f31600769c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f316c000f60 tx=0x7f316c007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.018+0000 7f31727fc640 1 -- 172.21.15.28:0/27304615 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f316805fc40 con 0x7f317415f7d0 2024-04-25T19:12:32.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:12:32.175 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:12:32.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.208+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31741f6e40 con 0x7f317415f7d0 2024-04-25T19:12:32.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.210+0000 7f31727fc640 1 -- 172.21.15.28:0/27304615 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18843 (secure 0 0 0) 0x7f316805f540 con 0x7f317415f7d0 2024-04-25T19:12:32.212 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:32.212 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:10.061001+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:11:27.072381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T19:11:28.257445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T19:11:31.947109+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.37","mappings":[{"from":7,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31600744b0 msgr2=0x7f31600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31600744b0 0x7f31600769c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f316c000f60 tx=0x7f316c007c00 comp rx=0 tx=0).stop 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 msgr2=0x7f31740ae270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f31740ae270 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f316800e870 tx=0x7f316800ed30 comp rx=0 tx=0).stop 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 shutdown_connections 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31600744b0 0x7f31600769c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31741f65c0 0x7f31740b34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f317415f7d0 0x7f31740ae270 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 --2- 172.21.15.28:0/27304615 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31740b5130 0x7f317415b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 >> 172.21.15.28:0/27304615 conn(0x7f31740b7ad0 msgr2=0x7f317415ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 shutdown_connections 2024-04-25T19:12:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.212+0000 7f317c096640 1 -- 172.21.15.28:0/27304615 wait complete. 2024-04-25T19:12:32.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T19:12:32.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.375+0000 7f83a6acd640 1 Processor -- start 2024-04-25T19:12:32.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.375+0000 7f83a6acd640 1 -- start start 2024-04-25T19:12:32.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.376+0000 7f83a6acd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a0160310 0x7f83a00d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.376+0000 7f83a6acd640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a0160720 con 0x7f83a00b9920 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.376+0000 7f83a6acd640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a00d5640 con 0x7f83a00b79f0 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.376+0000 7f83a6acd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f83a00d0f40 con 0x7f83a0160310 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.376+0000 7f83a506a640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f83a00b9920 0x7f83a00b9d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36870/0 (socket says 172.21.15.28:36870) 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.376+0000 7f83a506a640 1 -- 172.21.15.28:0/1872015602 learned_addr learned my addr 172.21.15.28:0/1872015602 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.376+0000 7f83a4869640 1 --2- 172.21.15.28:0/1872015602 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a0160310 0x7f83a00d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.376+0000 7f839f7fe640 1 -- 172.21.15.28:0/1872015602 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2003924665 0 0) 0x7f83a0160720 con 0x7f83a00b9920 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.377+0000 7f839f7fe640 1 -- 172.21.15.28:0/1872015602 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f838c003520 con 0x7f83a00b9920 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.377+0000 7f839f7fe640 1 -- 172.21.15.28:0/1872015602 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948706110 0 0) 0x7f83a00d5640 con 0x7f83a00b79f0 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.377+0000 7f839f7fe640 1 -- 172.21.15.28:0/1872015602 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83a0160720 con 0x7f83a00b79f0 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.377+0000 7f83a4869640 1 -- 172.21.15.28:0/1872015602 >> v1:172.21.15.53:6789/0 conn(0x7f83a00b79f0 legacy=0x7f83a00b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.377+0000 7f83a4869640 1 -- 172.21.15.28:0/1872015602 >> v1:172.21.15.28:6789/0 conn(0x7f83a00b9920 legacy=0x7f83a00b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.377+0000 7f839f7fe640 1 -- 172.21.15.28:0/1872015602 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 336656345 0 0) 0x7f838c003520 con 0x7f83a00b9920 2024-04-25T19:12:32.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.377+0000 7f83a4869640 1 -- 172.21.15.28:0/1872015602 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a00d1220 con 0x7f83a0160310 2024-04-25T19:12:32.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f83a4869640 1 --2- 172.21.15.28:0/1872015602 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a0160310 0x7f83a00d5100 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83880097d0 tx=0x7f838802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f880d8f1c4465cfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f839f7fe640 1 -- 172.21.15.28:0/1872015602 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8388037070 con 0x7f83a0160310 2024-04-25T19:12:32.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f839f7fe640 1 -- 172.21.15.28:0/1872015602 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f838802ac30 con 0x7f83a0160310 2024-04-25T19:12:32.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f839f7fe640 1 -- 172.21.15.28:0/1872015602 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83980019f0 con 0x7f83a0160310 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f83a6acd640 1 -- 172.21.15.28:0/1872015602 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a0160310 msgr2=0x7f83a00d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f83a6acd640 1 --2- 172.21.15.28:0/1872015602 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a0160310 0x7f83a00d5100 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f83880097d0 tx=0x7f838802a1a0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f83a6acd640 1 -- 172.21.15.28:0/1872015602 shutdown_connections 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f83a6acd640 1 --2- 172.21.15.28:0/1872015602 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a0160310 0x7f83a00d5100 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.378+0000 7f83a6acd640 1 -- 172.21.15.28:0/1872015602 >> 172.21.15.28:0/1872015602 conn(0x7f83a00d2a50 msgr2=0x7f83a00d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.379+0000 7f83a6acd640 1 -- 172.21.15.28:0/1872015602 shutdown_connections 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.379+0000 7f83a6acd640 1 -- 172.21.15.28:0/1872015602 wait complete. 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.380+0000 7f83a6acd640 1 Processor -- start 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.380+0000 7f83a6acd640 1 -- start start 2024-04-25T19:12:32.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a6acd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a00b79f0 0x7f83a01f5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a6acd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a00b9920 0x7f83a0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a6acd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83a0160310 0x7f83a0159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a6acd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83a00d0bf0 con 0x7f83a0160310 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a6acd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f83a00b9400 con 0x7f83a00b79f0 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a6acd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f83a00cc950 con 0x7f83a00b9920 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83a0160310 0x7f83a0159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83a0160310 0x7f83a0159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49340/0 (socket says 172.21.15.28:49340) 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 -- 172.21.15.28:0/3274621746 learned_addr learned my addr 172.21.15.28:0/3274621746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a4869640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a00b79f0 0x7f83a01f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f839ffff640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a00b9920 0x7f83a0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 -- 172.21.15.28:0/3274621746 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a00b9920 msgr2=0x7f83a0158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a00b9920 0x7f83a0158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 -- 172.21.15.28:0/3274621746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a00b79f0 msgr2=0x7f83a01f5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a00b79f0 0x7f83a01f5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 -- 172.21.15.28:0/3274621746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a0159eb0 con 0x7f83a0160310 2024-04-25T19:12:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.382+0000 7f83a506a640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83a0160310 0x7f83a0159370 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8398005c00 tx=0x7f839800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.383+0000 7f839d7fa640 1 -- 172.21.15.28:0/3274621746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398017070 con 0x7f83a0160310 2024-04-25T19:12:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.383+0000 7f839d7fa640 1 -- 172.21.15.28:0/3274621746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f839800ad60 con 0x7f83a0160310 2024-04-25T19:12:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.383+0000 7f839d7fa640 1 -- 172.21.15.28:0/3274621746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398012640 con 0x7f83a0160310 2024-04-25T19:12:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.383+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a0159bd0 con 0x7f83a0160310 2024-04-25T19:12:32.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.383+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a00c9a60 con 0x7f83a0160310 2024-04-25T19:12:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.384+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a0159940 con 0x7f83a0160310 2024-04-25T19:12:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.384+0000 7f839d7fa640 1 -- 172.21.15.28:0/3274621746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f839801a070 con 0x7f83a0160310 2024-04-25T19:12:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.385+0000 7f839d7fa640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83800744e0 0x7f83800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.385+0000 7f839d7fa640 1 -- 172.21.15.28:0/3274621746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8398095480 con 0x7f83a0160310 2024-04-25T19:12:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.385+0000 7f83a4869640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83800744e0 0x7f83800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.385+0000 7f83a4869640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83800744e0 0x7f83800769f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8388034000 tx=0x7f838802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.389+0000 7f839d7fa640 1 -- 172.21.15.28:0/3274621746 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f839805f1c0 con 0x7f83a0160310 2024-04-25T19:12:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.615+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f83a0050740 con 0x7f83a0160310 2024-04-25T19:12:32.618 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-25T19:12:32.616+0000 7fa80f29e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:12:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.617+0000 7f839d7fa640 1 -- 172.21.15.28:0/3274621746 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7f83980634c0 con 0x7f83a0160310 2024-04-25T19:12:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83800744e0 msgr2=0x7f83800769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83800744e0 0x7f83800769f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8388034000 tx=0x7f838802e000 comp rx=0 tx=0).stop 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83a0160310 msgr2=0x7f83a0159370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83a0160310 0x7f83a0159370 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8398005c00 tx=0x7f839800a430 comp rx=0 tx=0).stop 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 shutdown_connections 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f83800744e0 0x7f83800769f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83a0160310 0x7f83a0159370 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f83a00b9920 0x7f83a0158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 --2- 172.21.15.28:0/3274621746 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83a00b79f0 0x7f83a01f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 >> 172.21.15.28:0/3274621746 conn(0x7f83a00d2a50 msgr2=0x7f83a015e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 shutdown_connections 2024-04-25T19:12:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.620+0000 7f83a6acd640 1 -- 172.21.15.28:0/3274621746 wait complete. 2024-04-25T19:12:32.626 INFO:tasks.ceph.mds.d.smithi053.stderr: -5533> 2024-04-25T19:11:33.464+0000 7fc58dcd3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T19:12:32.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T19:12:32.684 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-04-25T19:12:32.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.746+0000 7fba67b34640 1 Processor -- start 2024-04-25T19:12:32.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.746+0000 7fba67b34640 1 -- start start 2024-04-25T19:12:32.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.747+0000 7fba67b34640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fba60160310 0x7fba600d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba67b34640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba600b79f0 0x7fba600b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba67b34640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba60160720 con 0x7fba6008cce0 2024-04-25T19:12:32.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba67b34640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fba600b8340 con 0x7fba600b79f0 2024-04-25T19:12:32.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba67b34640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fba600d5640 con 0x7fba60160310 2024-04-25T19:12:32.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba650cf640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba600b79f0 0x7fba600b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba660d1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fba6008cce0 0x7fba6008d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36880/0 (socket says 172.21.15.28:36880) 2024-04-25T19:12:32.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba660d1640 1 -- 172.21.15.28:0/635174669 learned_addr learned my addr 172.21.15.28:0/635174669 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:32.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba658d0640 1 --2- 172.21.15.28:0/635174669 >> v2:172.21.15.53:3300/0 conn(0x7fba60160310 0x7fba600d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba650cf640 1 -- 172.21.15.28:0/635174669 >> v2:172.21.15.53:3300/0 conn(0x7fba60160310 msgr2=0x7fba600d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:32.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba650cf640 1 --2- 172.21.15.28:0/635174669 >> v2:172.21.15.53:3300/0 conn(0x7fba60160310 0x7fba600d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba650cf640 1 -- 172.21.15.28:0/635174669 >> v1:172.21.15.28:6789/0 conn(0x7fba6008cce0 legacy=0x7fba6008d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba648ce640 1 -- 172.21.15.28:0/635174669 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1343372018 0 0) 0x7fba60160720 con 0x7fba6008cce0 2024-04-25T19:12:32.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.748+0000 7fba650cf640 1 -- 172.21.15.28:0/635174669 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba600d0f40 con 0x7fba600b79f0 2024-04-25T19:12:32.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.749+0000 7fba650cf640 1 --2- 172.21.15.28:0/635174669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba600b7e00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fba600d6be0 tx=0x7fba5402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d0ce8cb60dcc2f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.749+0000 7fba648ce640 1 -- 172.21.15.28:0/635174669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba54037070 con 0x7fba600b79f0 2024-04-25T19:12:32.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.749+0000 7fba648ce640 1 -- 172.21.15.28:0/635174669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba54032440 con 0x7fba600b79f0 2024-04-25T19:12:32.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.749+0000 7fba648ce640 1 -- 172.21.15.28:0/635174669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba5402ae60 con 0x7fba600b79f0 2024-04-25T19:12:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.750+0000 7fba67b34640 1 -- 172.21.15.28:0/635174669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 msgr2=0x7fba600b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.750+0000 7fba67b34640 1 --2- 172.21.15.28:0/635174669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba600b7e00 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fba600d6be0 tx=0x7fba5402a110 comp rx=0 tx=0).stop 2024-04-25T19:12:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.750+0000 7fba67b34640 1 -- 172.21.15.28:0/635174669 shutdown_connections 2024-04-25T19:12:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.750+0000 7fba67b34640 1 --2- 172.21.15.28:0/635174669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba600b7e00 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.750+0000 7fba67b34640 1 --2- 172.21.15.28:0/635174669 >> v2:172.21.15.53:3300/0 conn(0x7fba60160310 0x7fba600d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.750+0000 7fba67b34640 1 -- 172.21.15.28:0/635174669 >> 172.21.15.28:0/635174669 conn(0x7fba600d2a50 msgr2=0x7fba600d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.750+0000 7fba67b34640 1 -- 172.21.15.28:0/635174669 shutdown_connections 2024-04-25T19:12:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.750+0000 7fba67b34640 1 -- 172.21.15.28:0/635174669 wait complete. 2024-04-25T19:12:32.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.751+0000 7fba67b34640 1 Processor -- start 2024-04-25T19:12:32.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.752+0000 7fba67b34640 1 -- start start 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba67b34640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba6008cce0 0x7fba600cc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba67b34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba6015bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba67b34640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba60160310 0x7fba601e5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba67b34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba600b9400 con 0x7fba600b79f0 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba67b34640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fba600d1500 con 0x7fba60160310 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba67b34640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fba600d0bf0 con 0x7fba6008cce0 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba6015bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba6015bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49364/0 (socket says 172.21.15.28:49364) 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 -- 172.21.15.28:0/3477679448 learned_addr learned my addr 172.21.15.28:0/3477679448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba660d1640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba60160310 0x7fba601e5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba658d0640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba6008cce0 0x7fba600cc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 -- 172.21.15.28:0/3477679448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba6008cce0 msgr2=0x7fba600cc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba6008cce0 0x7fba600cc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 -- 172.21.15.28:0/3477679448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba60160310 msgr2=0x7fba601e5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba60160310 0x7fba601e5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 -- 172.21.15.28:0/3477679448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba601ea5e0 con 0x7fba600b79f0 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.753+0000 7fba650cf640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba6015bcf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fba54035040 tx=0x7fba540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.754+0000 7fba5e7fc640 1 -- 172.21.15.28:0/3477679448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba54037050 con 0x7fba600b79f0 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.754+0000 7fba5e7fc640 1 -- 172.21.15.28:0/3477679448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba54038040 con 0x7fba600b79f0 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.754+0000 7fba5e7fc640 1 -- 172.21.15.28:0/3477679448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba54004470 con 0x7fba600b79f0 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.754+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba600cadb0 con 0x7fba600b79f0 2024-04-25T19:12:32.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.754+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba600cb370 con 0x7fba600b79f0 2024-04-25T19:12:32.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.755+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba600d6390 con 0x7fba600b79f0 2024-04-25T19:12:32.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.755+0000 7fba5e7fc640 1 -- 172.21.15.28:0/3477679448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fba54003610 con 0x7fba600b79f0 2024-04-25T19:12:32.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.756+0000 7fba5e7fc640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fba4c0744e0 0x7fba4c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:32.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.756+0000 7fba5e7fc640 1 -- 172.21.15.28:0/3477679448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fba540b5db0 con 0x7fba600b79f0 2024-04-25T19:12:32.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.756+0000 7fba658d0640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fba4c0744e0 0x7fba4c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:32.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.756+0000 7fba658d0640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fba4c0744e0 0x7fba4c0769f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fba58009aa0 tx=0x7fba58009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:32.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.761+0000 7fba5e7fc640 1 -- 172.21.15.28:0/3477679448 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba5407fa90 con 0x7fba600b79f0 2024-04-25T19:12:32.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:32.993+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fba601ea370 con 0x7fba600b79f0 2024-04-25T19:12:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.621+0000 7fba5e7fc640 1 -- 172.21.15.28:0/3477679448 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7fba54083d90 con 0x7fba600b79f0 2024-04-25T19:12:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fba4c0744e0 msgr2=0x7fba4c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:33.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fba4c0744e0 0x7fba4c0769f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fba58009aa0 tx=0x7fba58009190 comp rx=0 tx=0).stop 2024-04-25T19:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 msgr2=0x7fba6015bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba6015bcf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fba54035040 tx=0x7fba540046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 shutdown_connections 2024-04-25T19:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fba4c0744e0 0x7fba4c0769f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba60160310 0x7fba601e5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba600b79f0 0x7fba6015bcf0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.624+0000 7fba67b34640 1 --2- 172.21.15.28:0/3477679448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba6008cce0 0x7fba600cc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.625+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 >> 172.21.15.28:0/3477679448 conn(0x7fba600d2a50 msgr2=0x7fba600d5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.625+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 shutdown_connections 2024-04-25T19:12:33.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.625+0000 7fba67b34640 1 -- 172.21.15.28:0/3477679448 wait complete. 2024-04-25T19:12:33.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T19:12:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.761+0000 7ff68c9a8640 1 Processor -- start 2024-04-25T19:12:33.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.761+0000 7ff68c9a8640 1 -- start start 2024-04-25T19:12:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.762+0000 7ff68c9a8640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff6880b47d0 0x7ff6881601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.762+0000 7ff68c9a8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6880d12e0 con 0x7ff68806e5e0 2024-04-25T19:12:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.762+0000 7ff68c9a8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6880d15c0 con 0x7ff6880d5690 2024-04-25T19:12:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.762+0000 7ff68c9a8640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7ff6880d4bb0 con 0x7ff6880b47d0 2024-04-25T19:12:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff686d9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff68806e5e0 0x7ff6880b9200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36886/0 (socket says 172.21.15.28:36886) 2024-04-25T19:12:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff686d9d640 1 -- 172.21.15.28:0/1340694201 learned_addr learned my addr 172.21.15.28:0/1340694201 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:33.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68659c640 1 --2- 172.21.15.28:0/1340694201 >> v2:172.21.15.53:3300/0 conn(0x7ff6880b47d0 0x7ff6881601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713204526 0 0) 0x7ff6880d12e0 con 0x7ff68806e5e0 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff670003520 con 0x7ff68806e5e0 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3615435441 0 0) 0x7ff670003520 con 0x7ff68806e5e0 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 >> v2:172.21.15.53:3300/0 conn(0x7ff6880b47d0 msgr2=0x7ff6881601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 --2- 172.21.15.28:0/1340694201 >> v2:172.21.15.53:3300/0 conn(0x7ff6880b47d0 0x7ff6881601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 >> v1:172.21.15.53:6789/0 conn(0x7ff6880d5690 legacy=0x7ff68806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6880d4e90 con 0x7ff68806e5e0 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7ff680001a10 con 0x7ff68806e5e0 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 >> v1:172.21.15.28:6789/0 conn(0x7ff68806e5e0 legacy=0x7ff6880b9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff670003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff670004800 0x7ff670004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff670005150 0x7ff67000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6880d4bb0 con 0x7ff6700039b0 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff67000c0c0 con 0x7ff670004800 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff67000c2c0 con 0x7ff670005150 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.763+0000 7ff68659c640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff670003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff685d9b640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff670004800 0x7ff670004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff686d9d640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff670005150 0x7ff67000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff68659c640 1 -- 172.21.15.28:0/1340694201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff670005150 msgr2=0x7ff67000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff68659c640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff670005150 0x7ff67000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff68659c640 1 -- 172.21.15.28:0/1340694201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff670004800 msgr2=0x7ff670004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff68659c640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff670004800 0x7ff670004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff68659c640 1 -- 172.21.15.28:0/1340694201 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff670003520 con 0x7ff6700039b0 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff686d9d640 1 -- 172.21.15.28:0/1340694201 reap_dead start 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.764+0000 7ff68659c640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff670003de0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a90 tx=0x7ff670005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ac958fa26f1084 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678036070 con 0x7ff6700039b0 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff678031400 con 0x7ff6700039b0 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68559a640 1 -- 172.21.15.28:0/1340694201 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678009690 con 0x7ff6700039b0 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68c9a8640 1 -- 172.21.15.28:0/1340694201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 msgr2=0x7ff670003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68c9a8640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff670003de0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a90 tx=0x7ff670005830 comp rx=0 tx=0).stop 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68c9a8640 1 -- 172.21.15.28:0/1340694201 shutdown_connections 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68c9a8640 1 --2- 172.21.15.28:0/1340694201 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff670003de0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68c9a8640 1 -- 172.21.15.28:0/1340694201 >> 172.21.15.28:0/1340694201 conn(0x7ff6880d2c60 msgr2=0x7ff6880d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68c9a8640 1 -- 172.21.15.28:0/1340694201 shutdown_connections 2024-04-25T19:12:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.765+0000 7ff68c9a8640 1 -- 172.21.15.28:0/1340694201 wait complete. 2024-04-25T19:12:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.767+0000 7ff68c9a8640 1 Processor -- start 2024-04-25T19:12:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.767+0000 7ff68c9a8640 1 -- start start 2024-04-25T19:12:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68c9a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff6880cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68c9a8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff688150430 0x7ff688150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68c9a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6880d06c0 0x7ff6880d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68c9a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6880b4e80 con 0x7ff6700039b0 2024-04-25T19:12:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68c9a8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7ff6880b4d00 con 0x7ff688150430 2024-04-25T19:12:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68c9a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7ff6880b5000 con 0x7ff6880d06c0 2024-04-25T19:12:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff6880cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff6880cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49374/0 (socket says 172.21.15.28:49374) 2024-04-25T19:12:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 -- 172.21.15.28:0/2433406344 learned_addr learned my addr 172.21.15.28:0/2433406344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff685d9b640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff688150430 0x7ff688150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff686d9d640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6880d06c0 0x7ff6880d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 -- 172.21.15.28:0/2433406344 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6880d06c0 msgr2=0x7ff6880d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6880d06c0 0x7ff6880d0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 -- 172.21.15.28:0/2433406344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff688150430 msgr2=0x7ff688150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff688150430 0x7ff688150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 -- 172.21.15.28:0/2433406344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6880d1010 con 0x7ff6700039b0 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68659c640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff6880cd860 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a90 tx=0x7ff678004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:33.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff676ffd640 1 -- 172.21.15.28:0/2433406344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678040020 con 0x7ff6700039b0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff676ffd640 1 -- 172.21.15.28:0/2433406344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff67803b680 con 0x7ff6700039b0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6881f9380 con 0x7ff6700039b0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff676ffd640 1 -- 172.21.15.28:0/2433406344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff678036040 con 0x7ff6700039b0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.769+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6881f9940 con 0x7ff6700039b0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.770+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff68806a100 con 0x7ff6700039b0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.771+0000 7ff676ffd640 1 -- 172.21.15.28:0/2433406344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff678043020 con 0x7ff6700039b0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.772+0000 7ff676ffd640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff6680744b0 0x7ff6680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.772+0000 7ff676ffd640 1 -- 172.21.15.28:0/2433406344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7ff6780b5e80 con 0x7ff6700039b0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.772+0000 7ff685d9b640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff6680744b0 0x7ff6680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.772+0000 7ff685d9b640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff6680744b0 0x7ff6680769c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff688153d10 tx=0x7ff67c007c50 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:33.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.776+0000 7ff676ffd640 1 -- 172.21.15.28:0/2433406344 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff67804a030 con 0x7ff6700039b0 2024-04-25T19:12:33.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:33.966+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff68808e890 con 0x7ff6700039b0 2024-04-25T19:12:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.622+0000 7ff676ffd640 1 -- 172.21.15.28:0/2433406344 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) ==== 181+0+0 (secure 0 0 0) 0x7ff6880d1010 con 0x7ff6700039b0 2024-04-25T19:12:34.625 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-04-25T19:12:34.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.626+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff6680744b0 msgr2=0x7ff6680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:34.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.626+0000 7ff68c9a8640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff6680744b0 0x7ff6680769c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff688153d10 tx=0x7ff67c007c50 comp rx=0 tx=0).stop 2024-04-25T19:12:34.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.626+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 msgr2=0x7ff6880cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:34.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.626+0000 7ff68c9a8640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff6880cd860 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff678009a90 tx=0x7ff678004730 comp rx=0 tx=0).stop 2024-04-25T19:12:34.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.627+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 shutdown_connections 2024-04-25T19:12:34.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.627+0000 7ff68c9a8640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff6680744b0 0x7ff6680769c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.627+0000 7ff68c9a8640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7ff6880d06c0 0x7ff6880d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.627+0000 7ff68c9a8640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff688150430 0x7ff688150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.627+0000 7ff68c9a8640 1 --2- 172.21.15.28:0/2433406344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6700039b0 0x7ff6880cd860 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.627+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 >> 172.21.15.28:0/2433406344 conn(0x7ff6880d2c60 msgr2=0x7ff6880cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:34.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.627+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 shutdown_connections 2024-04-25T19:12:34.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.627+0000 7ff68c9a8640 1 -- 172.21.15.28:0/2433406344 wait complete. 2024-04-25T19:12:34.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T19:12:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.763+0000 7f8d9ab6c640 1 Processor -- start 2024-04-25T19:12:34.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.763+0000 7f8d9ab6c640 1 -- start start 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.764+0000 7f8d9ab6c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d940b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.764+0000 7f8d9ab6c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d940cf250 0x7f8d940d1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.764+0000 7f8d9ab6c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d940b92e0 con 0x7f8d9409bb50 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.764+0000 7f8d9ab6c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8d9409d2a0 con 0x7f8d940cf250 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.764+0000 7f8d9ab6c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8d9409cfc0 con 0x7f8d9408cbb0 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.764+0000 7f8d99109640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d940cf250 0x7f8d940d1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.764+0000 7f8d99109640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d940cf250 0x7f8d940d1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49390/0 (socket says 172.21.15.28:49390) 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.764+0000 7f8d99109640 1 -- 172.21.15.28:0/372633628 learned_addr learned my addr 172.21.15.28:0/372633628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:34.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.765+0000 7f8d98908640 1 --2- 172.21.15.28:0/372633628 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d940b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.765+0000 7f8d99109640 1 -- 172.21.15.28:0/372633628 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 msgr2=0x7f8d940b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.765+0000 7f8d99109640 1 --2- 172.21.15.28:0/372633628 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d940b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.765+0000 7f8d99109640 1 -- 172.21.15.28:0/372633628 >> v1:172.21.15.53:6789/0 conn(0x7f8d9409bb50 legacy=0x7f8d9409bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.765+0000 7f8d99109640 1 -- 172.21.15.28:0/372633628 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9406d9c0 con 0x7f8d940cf250 2024-04-25T19:12:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.765+0000 7f8d98908640 1 --2- 172.21.15.28:0/372633628 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d940b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d99109640 1 --2- 172.21.15.28:0/372633628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d940cf250 0x7f8d940d1b80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009870 tx=0x7f8d8402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7edda26e96318b77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d937fe640 1 -- 172.21.15.28:0/372633628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84037070 con 0x7f8d940cf250 2024-04-25T19:12:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d937fe640 1 -- 172.21.15.28:0/372633628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d84032440 con 0x7f8d940cf250 2024-04-25T19:12:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d937fe640 1 -- 172.21.15.28:0/372633628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d84032810 con 0x7f8d940cf250 2024-04-25T19:12:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/372633628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d940cf250 msgr2=0x7f8d940d1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/372633628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d940cf250 0x7f8d940d1b80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f8d84009870 tx=0x7f8d8402a1c0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/372633628 shutdown_connections 2024-04-25T19:12:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/372633628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d940cf250 0x7f8d940d1b80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/372633628 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d940b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/372633628 >> 172.21.15.28:0/372633628 conn(0x7f8d940b5470 msgr2=0x7f8d940b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/372633628 shutdown_connections 2024-04-25T19:12:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.766+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/372633628 wait complete. 2024-04-25T19:12:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.768+0000 7f8d9ab6c640 1 Processor -- start 2024-04-25T19:12:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.768+0000 7f8d9ab6c640 1 -- start start 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d9ab6c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d941dac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d9ab6c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9409bb50 0x7f8d940ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d9ab6c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d940cf250 0x7f8d941e9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d9ab6c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d940b4450 con 0x7f8d9409bb50 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d9ab6c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8d940b42d0 con 0x7f8d940cf250 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d9ab6c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8d940d20c0 con 0x7f8d9408cbb0 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9409bb50 0x7f8d940ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9409bb50 0x7f8d940ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49402/0 (socket says 172.21.15.28:49402) 2024-04-25T19:12:34.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 -- 172.21.15.28:0/4272614005 learned_addr learned my addr 172.21.15.28:0/4272614005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d98908640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d941dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d99109640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d940cf250 0x7f8d941e9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 -- 172.21.15.28:0/4272614005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 msgr2=0x7f8d941dac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d941dac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 -- 172.21.15.28:0/4272614005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d940cf250 msgr2=0x7f8d941e9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d940cf250 0x7f8d941e9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 -- 172.21.15.28:0/4272614005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d940b2560 con 0x7f8d9409bb50 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.770+0000 7f8d93fff640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9409bb50 0x7f8d940ca0d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800c3c0 tx=0x7f8d8800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.771+0000 7f8d917fa640 1 -- 172.21.15.28:0/4272614005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88017070 con 0x7f8d9409bb50 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.771+0000 7f8d917fa640 1 -- 172.21.15.28:0/4272614005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d880052d0 con 0x7f8d9409bb50 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.771+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d940b2840 con 0x7f8d9409bb50 2024-04-25T19:12:34.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.771+0000 7f8d917fa640 1 -- 172.21.15.28:0/4272614005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88003830 con 0x7f8d9409bb50 2024-04-25T19:12:34.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.771+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d940b2e00 con 0x7f8d9409bb50 2024-04-25T19:12:34.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.772+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d940b22d0 con 0x7f8d9409bb50 2024-04-25T19:12:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.773+0000 7f8d917fa640 1 -- 172.21.15.28:0/4272614005 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8d8801a070 con 0x7f8d9409bb50 2024-04-25T19:12:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.773+0000 7f8d917fa640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d800744b0 0x7f8d800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.773+0000 7f8d917fa640 1 -- 172.21.15.28:0/4272614005 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6383+0+0 (secure 0 0 0) 0x7f8d880946b0 con 0x7f8d9409bb50 2024-04-25T19:12:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.773+0000 7f8d98908640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d800744b0 0x7f8d800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.774+0000 7f8d98908640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d800744b0 0x7f8d800769c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8d9409c7c0 tx=0x7f8d8c009870 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.778+0000 7f8d917fa640 1 -- 172.21.15.28:0/4272614005 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d8805e550 con 0x7f8d9409bb50 2024-04-25T19:12:34.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:34.970+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8d9406a100 con 0x7f8d9409bb50 2024-04-25T19:12:35.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.627+0000 7f8d917fa640 1 -- 172.21.15.28:0/4272614005 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) ==== 169+0+0 (secure 0 0 0) 0x7f8d88062850 con 0x7f8d9409bb50 2024-04-25T19:12:35.630 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-04-25T19:12:35.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d800744b0 msgr2=0x7f8d800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d800744b0 0x7f8d800769c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8d9409c7c0 tx=0x7f8d8c009870 comp rx=0 tx=0).stop 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9409bb50 msgr2=0x7f8d940ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9409bb50 0x7f8d940ca0d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8d8800c3c0 tx=0x7f8d8800c880 comp rx=0 tx=0).stop 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 shutdown_connections 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d800744b0 0x7f8d800769c0 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8d9409c7c0 tx=0x7f8d8c009870 comp rx=0 tx=0).stop 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d940cf250 0x7f8d941e9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d9409bb50 0x7f8d940ca0d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 --2- 172.21.15.28:0/4272614005 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d9408cbb0 0x7f8d941dac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.634+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 >> 172.21.15.28:0/4272614005 conn(0x7f8d940b5470 msgr2=0x7f8d94068740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.635+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 shutdown_connections 2024-04-25T19:12:35.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.636+0000 7f8d9ab6c640 1 -- 172.21.15.28:0/4272614005 wait complete. 2024-04-25T19:12:35.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T19:12:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.774+0000 7fc34179f640 1 Processor -- start 2024-04-25T19:12:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.774+0000 7fc34179f640 1 -- start start 2024-04-25T19:12:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc34179f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c0bb2f0 0x7fc33c06e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc34179f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc33c06e540 0x7fc33c15f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc34179f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc33c06e950 con 0x7fc33c0d1150 2024-04-25T19:12:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc34179f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fc33c0b9260 con 0x7fc33c06e540 2024-04-25T19:12:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc34179f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc33c0d1560 con 0x7fc33c0bb2f0 2024-04-25T19:12:35.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc33b7fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c0bb2f0 0x7fc33c06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc340f9e640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc33c06e540 0x7fc33c15f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc33b7fe640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c0bb2f0 0x7fc33c06e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:44702/0 (socket says 172.21.15.28:44702) 2024-04-25T19:12:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc33b7fe640 1 -- 172.21.15.28:0/1628401095 learned_addr learned my addr 172.21.15.28:0/1628401095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc340f9e640 1 -- 172.21.15.28:0/1628401095 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c0bb2f0 msgr2=0x7fc33c06e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc340f9e640 1 --2- 172.21.15.28:0/1628401095 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c0bb2f0 0x7fc33c06e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc340f9e640 1 -- 172.21.15.28:0/1628401095 >> v1:172.21.15.53:6789/0 conn(0x7fc33c0d1150 legacy=0x7fc33c0d4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc340f9e640 1 -- 172.21.15.28:0/1628401095 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33c0b8f30 con 0x7fc33c06e540 2024-04-25T19:12:35.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.776+0000 7fc33b7fe640 1 --2- 172.21.15.28:0/1628401095 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c0bb2f0 0x7fc33c06e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.777+0000 7fc340f9e640 1 --2- 172.21.15.28:0/1628401095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c06e540 0x7fc33c15f880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc334009d90 tx=0x7fc33402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f63f60859cfc30 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.777+0000 7fc33affd640 1 -- 172.21.15.28:0/1628401095 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334031600 con 0x7fc33c06e540 2024-04-25T19:12:35.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.777+0000 7fc33affd640 1 -- 172.21.15.28:0/1628401095 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3340317a0 con 0x7fc33c06e540 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.777+0000 7fc33affd640 1 -- 172.21.15.28:0/1628401095 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334031b70 con 0x7fc33c06e540 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.778+0000 7fc34179f640 1 -- 172.21.15.28:0/1628401095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c06e540 msgr2=0x7fc33c15f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.778+0000 7fc34179f640 1 --2- 172.21.15.28:0/1628401095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c06e540 0x7fc33c15f880 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc334009d90 tx=0x7fc33402a5d0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.778+0000 7fc34179f640 1 -- 172.21.15.28:0/1628401095 shutdown_connections 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.778+0000 7fc34179f640 1 --2- 172.21.15.28:0/1628401095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c06e540 0x7fc33c15f880 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.778+0000 7fc34179f640 1 --2- 172.21.15.28:0/1628401095 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c0bb2f0 0x7fc33c06e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.778+0000 7fc34179f640 1 -- 172.21.15.28:0/1628401095 >> 172.21.15.28:0/1628401095 conn(0x7fc33c06d170 msgr2=0x7fc33c0d24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.778+0000 7fc34179f640 1 -- 172.21.15.28:0/1628401095 shutdown_connections 2024-04-25T19:12:35.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.778+0000 7fc34179f640 1 -- 172.21.15.28:0/1628401095 wait complete. 2024-04-25T19:12:35.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.780+0000 7fc34179f640 1 Processor -- start 2024-04-25T19:12:35.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.780+0000 7fc34179f640 1 -- start start 2024-04-25T19:12:35.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.781+0000 7fc34179f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c06e540 0x7fc33c153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:35.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.781+0000 7fc34179f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c0bb2f0 0x7fc33c0cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.781+0000 7fc34179f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc33c0d1150 0x7fc33c1f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.781+0000 7fc34179f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c0d4ec0 con 0x7fc33c0d1150 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.781+0000 7fc34179f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc33c0d4be0 con 0x7fc33c0bb2f0 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.781+0000 7fc34179f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc33c0d51a0 con 0x7fc33c06e540 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc33c0d1150 0x7fc33c1f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc33c0d1150 0x7fc33c1f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49410/0 (socket says 172.21.15.28:49410) 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 -- 172.21.15.28:0/584530670 learned_addr learned my addr 172.21.15.28:0/584530670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc33b7fe640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c0bb2f0 0x7fc33c0cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc33bfff640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c06e540 0x7fc33c153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 -- 172.21.15.28:0/584530670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c06e540 msgr2=0x7fc33c153ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c06e540 0x7fc33c153ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 -- 172.21.15.28:0/584530670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c0bb2f0 msgr2=0x7fc33c0cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:35.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c0bb2f0 0x7fc33c0cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 -- 172.21.15.28:0/584530670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33c1543d0 con 0x7fc33c0d1150 2024-04-25T19:12:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc340f9e640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc33c0d1150 0x7fc33c1f5bf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc33402abc0 tx=0x7fc334004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc338ff9640 1 -- 172.21.15.28:0/584530670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33403f070 con 0x7fc33c0d1150 2024-04-25T19:12:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc338ff9640 1 -- 172.21.15.28:0/584530670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc334030b60 con 0x7fc33c0d1150 2024-04-25T19:12:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc33c1546b0 con 0x7fc33c0d1150 2024-04-25T19:12:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc338ff9640 1 -- 172.21.15.28:0/584530670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc334009450 con 0x7fc33c0d1150 2024-04-25T19:12:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.782+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33c154c70 con 0x7fc33c0d1150 2024-04-25T19:12:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.784+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc33c154eb0 con 0x7fc33c0d1150 2024-04-25T19:12:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.784+0000 7fc338ff9640 1 -- 172.21.15.28:0/584530670 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc33403b410 con 0x7fc33c0d1150 2024-04-25T19:12:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.785+0000 7fc338ff9640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc3280744b0 0x7fc3280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.785+0000 7fc33bfff640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc3280744b0 0x7fc3280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.785+0000 7fc338ff9640 1 -- 172.21.15.28:0/584530670 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5853+0+0 (secure 0 0 0) 0x7fc3340b67c0 con 0x7fc33c0d1150 2024-04-25T19:12:35.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.785+0000 7fc33bfff640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc3280744b0 0x7fc3280769c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc33c0d55e0 tx=0x7fc330009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:35.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.790+0000 7fc338ff9640 1 -- 172.21.15.28:0/584530670 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc33c154eb0 con 0x7fc33c0d1150 2024-04-25T19:12:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:35.975+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc33c1565b0 con 0x7fc33c0d1150 2024-04-25T19:12:36.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.633+0000 7fc338ff9640 1 -- 172.21.15.28:0/584530670 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) ==== 178+0+0 (secure 0 0 0) 0x7fc3340807f0 con 0x7fc33c0d1150 2024-04-25T19:12:36.635 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data_ec' removed 2024-04-25T19:12:36.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc3280744b0 msgr2=0x7fc3280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:36.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc3280744b0 0x7fc3280769c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc33c0d55e0 tx=0x7fc330009040 comp rx=0 tx=0).stop 2024-04-25T19:12:36.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc33c0d1150 msgr2=0x7fc33c1f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc33c0d1150 0x7fc33c1f5bf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc33402abc0 tx=0x7fc334004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 shutdown_connections 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc3280744b0 0x7fc3280769c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc33c0d1150 0x7fc33c1f5bf0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc33c0bb2f0 0x7fc33c0cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 --2- 172.21.15.28:0/584530670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc33c06e540 0x7fc33c153ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 >> 172.21.15.28:0/584530670 conn(0x7fc33c06d170 msgr2=0x7fc33c0b4640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 shutdown_connections 2024-04-25T19:12:36.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.635+0000 7fc34179f640 1 -- 172.21.15.28:0/584530670 wait complete. 2024-04-25T19:12:36.650 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T19:12:36.650 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T19:12:36.650 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T19:12:36.650 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:12:36.650 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T19:12:36.651 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T19:12:36.651 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T19:12:36.651 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:12:36.651 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T19:12:36.651 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T19:12:36.651 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T19:12:36.652 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:12:36.652 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T19:12:36.652 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T19:12:36.652 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T19:12:36.652 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:12:36.661 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T19:12:36.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T19:12:36.662 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T19:12:36.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T19:12:36.666 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T19:12:36.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T19:12:36.668 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T19:12:36.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T19:12:36.704 INFO:tasks.ceph.mds.c:Started 2024-04-25T19:12:36.704 DEBUG:teuthology.parallel:result is None 2024-04-25T19:12:36.705 INFO:tasks.ceph.mds.a:Started 2024-04-25T19:12:36.705 DEBUG:teuthology.parallel:result is None 2024-04-25T19:12:36.710 INFO:tasks.ceph.mds.b:Started 2024-04-25T19:12:36.711 DEBUG:teuthology.parallel:result is None 2024-04-25T19:12:36.712 INFO:tasks.ceph.mds.d:Started 2024-04-25T19:12:36.712 DEBUG:teuthology.parallel:result is None 2024-04-25T19:12:36.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T19:12:36.793 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-04-25T19:12:36.794 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-04-25T19:12:36.804 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-04-25T19:12:36.805 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-04-25T19:12:36.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.830+0000 7f94b841e640 1 Processor -- start 2024-04-25T19:12:36.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.830+0000 7f94b841e640 1 -- start start 2024-04-25T19:12:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b841e640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f94b00b1a90 0x7f94b00cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b841e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b00cff50 con 0x7f94b00b3080 2024-04-25T19:12:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b841e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b00b1f90 con 0x7f94b00d20d0 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b841e640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f94b00b22c0 con 0x7f94b00b1a90 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b59b9640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f94b00b3080 0x7f94b00d19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36894/0 (socket says 172.21.15.28:36894) 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b59b9640 1 -- 172.21.15.28:0/352328386 learned_addr learned my addr 172.21.15.28:0/352328386 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b61ba640 1 --2- 172.21.15.28:0/352328386 >> v2:172.21.15.53:3300/0 conn(0x7f94b00b1a90 0x7f94b00cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b51b8640 1 -- 172.21.15.28:0/352328386 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4004151286 0 0) 0x7f94b00cff50 con 0x7f94b00b3080 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b51b8640 1 -- 172.21.15.28:0/352328386 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9498003520 con 0x7f94b00b3080 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b51b8640 1 -- 172.21.15.28:0/352328386 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2913392687 0 0) 0x7f94b00b1f90 con 0x7f94b00d20d0 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b51b8640 1 -- 172.21.15.28:0/352328386 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94b00cff50 con 0x7f94b00d20d0 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b61ba640 1 -- 172.21.15.28:0/352328386 >> v1:172.21.15.53:6789/0 conn(0x7f94b00d20d0 legacy=0x7f94b00b3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b61ba640 1 -- 172.21.15.28:0/352328386 >> v1:172.21.15.28:6789/0 conn(0x7f94b00b3080 legacy=0x7f94b00d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.832+0000 7f94b61ba640 1 -- 172.21.15.28:0/352328386 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b00b25a0 con 0x7f94b00b1a90 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.833+0000 7f94b61ba640 1 --2- 172.21.15.28:0/352328386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 0x7f94b00cf3e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f94a80097a0 tx=0x7f94a802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cadf89cc8824db33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:36.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.833+0000 7f94b51b8640 1 -- 172.21.15.28:0/352328386 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8037070 con 0x7f94b00b1a90 2024-04-25T19:12:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.833+0000 7f94b51b8640 1 -- 172.21.15.28:0/352328386 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a8032440 con 0x7f94b00b1a90 2024-04-25T19:12:36.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.833+0000 7f94b51b8640 1 -- 172.21.15.28:0/352328386 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8032810 con 0x7f94b00b1a90 2024-04-25T19:12:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.834+0000 7f94b841e640 1 -- 172.21.15.28:0/352328386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 msgr2=0x7f94b00cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.834+0000 7f94b841e640 1 --2- 172.21.15.28:0/352328386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 0x7f94b00cf3e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f94a80097a0 tx=0x7f94a802a1a0 comp rx=0 tx=0).stop 2024-04-25T19:12:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.834+0000 7f94b841e640 1 -- 172.21.15.28:0/352328386 shutdown_connections 2024-04-25T19:12:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.834+0000 7f94b841e640 1 --2- 172.21.15.28:0/352328386 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 0x7f94b00cf3e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.834+0000 7f94b841e640 1 -- 172.21.15.28:0/352328386 >> 172.21.15.28:0/352328386 conn(0x7f94b00b7570 msgr2=0x7f94b00b7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.834+0000 7f94b841e640 1 -- 172.21.15.28:0/352328386 shutdown_connections 2024-04-25T19:12:36.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.834+0000 7f94b841e640 1 -- 172.21.15.28:0/352328386 wait complete. 2024-04-25T19:12:36.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.836+0000 7f94b841e640 1 Processor -- start 2024-04-25T19:12:36.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.836+0000 7f94b841e640 1 -- start start 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b841e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 0x7f94b01eb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b841e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94b00b3080 0x7f94b015fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b841e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94b00d20d0 0x7f94b015ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b841e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94b0164940 con 0x7f94b00d20d0 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b841e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f94b00b2b60 con 0x7f94b00b1a90 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b841e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f94b0164ac0 con 0x7f94b00b3080 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b69bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94b00d20d0 0x7f94b015ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b69bb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94b00d20d0 0x7f94b015ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49462/0 (socket says 172.21.15.28:49462) 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.837+0000 7f94b69bb640 1 -- 172.21.15.28:0/2289073293 learned_addr learned my addr 172.21.15.28:0/2289073293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b61ba640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 0x7f94b01eb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:36.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b59b9640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94b00b3080 0x7f94b015fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b69bb640 1 -- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94b00b3080 msgr2=0x7f94b015fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b69bb640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94b00b3080 0x7f94b015fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b69bb640 1 -- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 msgr2=0x7f94b01eb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b69bb640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 0x7f94b01eb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b69bb640 1 -- 172.21.15.28:0/2289073293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b015ffa0 con 0x7f94b00d20d0 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b61ba640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 0x7f94b01eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b69bb640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94b00d20d0 0x7f94b015ed20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f94a000c6e0 tx=0x7f94a000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94aeffd640 1 -- 172.21.15.28:0/2289073293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0003b70 con 0x7f94b00d20d0 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94aeffd640 1 -- 172.21.15.28:0/2289073293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a0003d10 con 0x7f94b00d20d0 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b01ea140 con 0x7f94b00d20d0 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94aeffd640 1 -- 172.21.15.28:0/2289073293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a00052d0 con 0x7f94b00d20d0 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.838+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b01ea700 con 0x7f94b00d20d0 2024-04-25T19:12:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.839+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94b006ec80 con 0x7f94b00d20d0 2024-04-25T19:12:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.840+0000 7f94aeffd640 1 -- 172.21.15.28:0/2289073293 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f94a0005470 con 0x7f94b00d20d0 2024-04-25T19:12:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.840+0000 7f94aeffd640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f949c0744b0 0x7f949c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.840+0000 7f94aeffd640 1 -- 172.21.15.28:0/2289073293 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5335+0+0 (secure 0 0 0) 0x7f94a0095040 con 0x7f94b00d20d0 2024-04-25T19:12:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.840+0000 7f94b61ba640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f949c0744b0 0x7f949c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.841+0000 7f94b61ba640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f949c0744b0 0x7f949c0769c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f94a8034000 tx=0x7f94a802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:36.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:36.845+0000 7f94aeffd640 1 -- 172.21.15.28:0/2289073293 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94a005f8a0 con 0x7f94b00d20d0 2024-04-25T19:12:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.041+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f94b00cfa10 con 0x7f94b00d20d0 2024-04-25T19:12:37.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.643+0000 7f94aeffd640 1 -- 172.21.15.28:0/2289073293 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) ==== 97+0+0 (secure 0 0 0) 0x7f94a005f1a0 con 0x7f94b00d20d0 2024-04-25T19:12:37.645 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-04-25T19:12:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.646+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f949c0744b0 msgr2=0x7f949c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.646+0000 7f94b841e640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f949c0744b0 0x7f949c0769c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f94a8034000 tx=0x7f94a802e000 comp rx=0 tx=0).stop 2024-04-25T19:12:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.646+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94b00d20d0 msgr2=0x7f94b015ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.646+0000 7f94b841e640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94b00d20d0 0x7f94b015ed20 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f94a000c6e0 tx=0x7f94a000cba0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.647+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 shutdown_connections 2024-04-25T19:12:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.647+0000 7f94b841e640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f949c0744b0 0x7f949c0769c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.647+0000 7f94b841e640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94b00d20d0 0x7f94b015ed20 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.647+0000 7f94b841e640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f94b00b3080 0x7f94b015fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.647+0000 7f94b841e640 1 --2- 172.21.15.28:0/2289073293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94b00b1a90 0x7f94b01eb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.647+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 >> 172.21.15.28:0/2289073293 conn(0x7f94b00b7570 msgr2=0x7f94b0071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:37.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.647+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 shutdown_connections 2024-04-25T19:12:37.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.647+0000 7f94b841e640 1 -- 172.21.15.28:0/2289073293 wait complete. 2024-04-25T19:12:37.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T19:12:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.783+0000 7fb695cb1640 1 Processor -- start 2024-04-25T19:12:37.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.783+0000 7fb695cb1640 1 -- start start 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb695cb1640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb69009cb20 0x7fb6900b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb695cb1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb690163850 0x7fb6900ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb695cb1640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6900d1d30 con 0x7fb6900ce550 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb695cb1640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb6900d2620 con 0x7fb69009cb20 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb695cb1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb69006ebb0 con 0x7fb690163850 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68f7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb69009cb20 0x7fb6900b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb6900ce550 0x7fb6900d2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36906/0 (socket says 172.21.15.28:36906) 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68ffff640 1 -- 172.21.15.28:0/2989591319 learned_addr learned my addr 172.21.15.28:0/2989591319 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:37.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68effd640 1 --2- 172.21.15.28:0/2989591319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb690163850 0x7fb6900ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68f7fe640 1 -- 172.21.15.28:0/2989591319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb690163850 msgr2=0x7fb6900ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68f7fe640 1 --2- 172.21.15.28:0/2989591319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb690163850 0x7fb6900ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68f7fe640 1 -- 172.21.15.28:0/2989591319 >> v1:172.21.15.28:6789/0 conn(0x7fb6900ce550 legacy=0x7fb6900d2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68f7fe640 1 -- 172.21.15.28:0/2989591319 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69006e5f0 con 0x7fb69009cb20 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68e7fc640 1 -- 172.21.15.28:0/2989591319 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2770056959 0 0) 0x7fb6900d1d30 con 0x7fb6900ce550 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.785+0000 7fb68effd640 1 --2- 172.21.15.28:0/2989591319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb690163850 0x7fb6900ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.786+0000 7fb68f7fe640 1 --2- 172.21.15.28:0/2989591319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb69009cb20 0x7fb6900b91a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb6800097a0 tx=0x7fb68002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6219a9477f4164 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.786+0000 7fb68e7fc640 1 -- 172.21.15.28:0/2989591319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb680037070 con 0x7fb69009cb20 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.786+0000 7fb68e7fc640 1 -- 172.21.15.28:0/2989591319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb680032440 con 0x7fb69009cb20 2024-04-25T19:12:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.786+0000 7fb68e7fc640 1 -- 172.21.15.28:0/2989591319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb680032810 con 0x7fb69009cb20 2024-04-25T19:12:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.787+0000 7fb695cb1640 1 -- 172.21.15.28:0/2989591319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb69009cb20 msgr2=0x7fb6900b91a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.787+0000 7fb695cb1640 1 --2- 172.21.15.28:0/2989591319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb69009cb20 0x7fb6900b91a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb6800097a0 tx=0x7fb68002a180 comp rx=0 tx=0).stop 2024-04-25T19:12:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.787+0000 7fb695cb1640 1 -- 172.21.15.28:0/2989591319 shutdown_connections 2024-04-25T19:12:37.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.787+0000 7fb695cb1640 1 --2- 172.21.15.28:0/2989591319 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb690163850 0x7fb6900ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.787+0000 7fb695cb1640 1 --2- 172.21.15.28:0/2989591319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb69009cb20 0x7fb6900b91a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.787+0000 7fb695cb1640 1 -- 172.21.15.28:0/2989591319 >> 172.21.15.28:0/2989591319 conn(0x7fb6900b54b0 msgr2=0x7fb6900b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.787+0000 7fb695cb1640 1 -- 172.21.15.28:0/2989591319 shutdown_connections 2024-04-25T19:12:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.787+0000 7fb695cb1640 1 -- 172.21.15.28:0/2989591319 wait complete. 2024-04-25T19:12:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.789+0000 7fb695cb1640 1 Processor -- start 2024-04-25T19:12:37.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.789+0000 7fb695cb1640 1 -- start start 2024-04-25T19:12:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb695cb1640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb69009cb20 0x7fb690158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb695cb1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6900ce550 0x7fb6900b4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb695cb1640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb690163850 0x7fb6900b7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:37.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb695cb1640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6900d22b0 con 0x7fb6900ce550 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb695cb1640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6900d1fd0 con 0x7fb690163850 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb695cb1640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb690163c60 con 0x7fb69009cb20 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb68effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6900ce550 0x7fb6900b4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb68effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6900ce550 0x7fb6900b4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49478/0 (socket says 172.21.15.28:49478) 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb68effd640 1 -- 172.21.15.28:0/3345035195 learned_addr learned my addr 172.21.15.28:0/3345035195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.790+0000 7fb68f7fe640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb69009cb20 0x7fb690158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb68ffff640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb690163850 0x7fb6900b7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb68effd640 1 -- 172.21.15.28:0/3345035195 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb69009cb20 msgr2=0x7fb690158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb68effd640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb69009cb20 0x7fb690158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb68effd640 1 -- 172.21.15.28:0/3345035195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb690163850 msgr2=0x7fb6900b7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb68effd640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb690163850 0x7fb6900b7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb68effd640 1 -- 172.21.15.28:0/3345035195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb690158bc0 con 0x7fb6900ce550 2024-04-25T19:12:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb68effd640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6900ce550 0x7fb6900b4140 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c820 tx=0x7fb68400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb677fff640 1 -- 172.21.15.28:0/3345035195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684015070 con 0x7fb6900ce550 2024-04-25T19:12:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb677fff640 1 -- 172.21.15.28:0/3345035195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb684007a30 con 0x7fb6900ce550 2024-04-25T19:12:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb677fff640 1 -- 172.21.15.28:0/3345035195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684007de0 con 0x7fb6900ce550 2024-04-25T19:12:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6900b4dc0 con 0x7fb6900ce550 2024-04-25T19:12:37.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.791+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb690159d20 con 0x7fb6900ce550 2024-04-25T19:12:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.792+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6900b4b30 con 0x7fb6900ce550 2024-04-25T19:12:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.793+0000 7fb677fff640 1 -- 172.21.15.28:0/3345035195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb684018070 con 0x7fb6900ce550 2024-04-25T19:12:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.793+0000 7fb677fff640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb67c0744b0 0x7fb67c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.794+0000 7fb677fff640 1 -- 172.21.15.28:0/3345035195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fb684094fe0 con 0x7fb6900ce550 2024-04-25T19:12:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.794+0000 7fb68f7fe640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb67c0744b0 0x7fb67c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:37.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.794+0000 7fb68f7fe640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb67c0744b0 0x7fb67c0769c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb680034000 tx=0x7fb68002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:37.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:37.798+0000 7fb677fff640 1 -- 172.21.15.28:0/3345035195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb690158bc0 con 0x7fb6900ce550 2024-04-25T19:12:38.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.056+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb690050740 con 0x7fb6900ce550 2024-04-25T19:12:38.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.057+0000 7fb677fff640 1 -- 172.21.15.28:0/3345035195 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7fb68405f920 con 0x7fb6900ce550 2024-04-25T19:12:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:38.059 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:12:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-04-25T19:12:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-04-25T19:12:38.059 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCqqipmiMziIRAAIlkDemOas1/wLLSfrPOtSw==", 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-04-25T19:12:38.060 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCqqipmSLzKLhAAlKSU261QzMzYnLfxJDAh6w==", 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCqqipmDDFcKRAA/e20s2LVvRm6r8j/elUESg==", 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCqqipmOCFsNhAAFpeHA6Rfn9noLEk/t2iS7Q==", 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-04-25T19:12:38.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCsqipmfYl4GxAAhlhjSJHAr07miHRZvkrW/w==", 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCvqipmgLkkGxAAKnKUs3oTOfZDKGe+Bd7MZg==", 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-04-25T19:12:38.064 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCyqipm6RbtGRAA/vItjrpYyYv98tVL1xDwHw==", 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQC1qipmfYO1GBAA33Hurpq9u03VKZk2Hw99LA==", 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQC5qipmN5mHIhAAxNKhJie7i/i1ENVDbLN2/g==", 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQC8qipmOgu1IRAAnUsdR58QJkdmI71ugbA/8Q==", 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQC/qipmbnihIRAA5M1YN90/yOeRMaDt6QW3Fg==", 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-04-25T19:12:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDCqipm1v9sIBAAEUE5Th5yvG3g2uVmioB2rw==", 2024-04-25T19:12:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-04-25T19:12:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-04-25T19:12:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCqqipmM9YPOxAAJwNJw9dfCED4wy79hvmLNg==", 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-04-25T19:12:38.070 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCrqipm5bLpBhAASSZf55x8RA98ZzfDf8pocw==", 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-04-25T19:12:38.071 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCrqipmv7UoAxAAMsoPXb/JY3YHQPtO+QE6Ew==", 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-04-25T19:12:38.072 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCrqipmEWGTChAAso9pw1SJnc/t9bczpddQJg==", 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-04-25T19:12:38.073 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCqqipmVQBTBBAACf2rgDYN8G8x/Hw5rdrmAQ==", 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-04-25T19:12:38.074 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mds", 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDIqipmgnNHIRAAnfwyhTxwG5sIc7yQy8cI3g==", 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.075 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mds" 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mgr", 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDIqipmPnxHIRAAgG2LIhGq4JNyoIXtxv8huQ==", 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-osd", 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDIqipmx4NHIRAAFswliRLsvYHRV0cqRBdKyQ==", 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-osd" 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd", 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDIqipm/IpHIRAADlAsmaXVUs5xxhjDeKUPsQ==", 2024-04-25T19:12:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDIqipmtJJHIRAA4BI11Jkb8JuqLrLco+MHiw==", 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rgw", 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQDIqipma5pHIRAAhI6lBdr4dJdJPdJuezSjGA==", 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCqqipmt2IdHhAAU7Iv0W1CnFpsi+zs3Zg2Yg==", 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQCqqipm7QHJGRAAsIrA+E/sN4HzAyzh3RIabg==", 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-25T19:12:38.081 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:12:38.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb67c0744b0 msgr2=0x7fb67c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb67c0744b0 0x7fb67c0769c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb680034000 tx=0x7fb68002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6900ce550 msgr2=0x7fb6900b4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6900ce550 0x7fb6900b4140 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb68400c820 tx=0x7fb68400cce0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 shutdown_connections 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb67c0744b0 0x7fb67c0769c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb690163850 0x7fb6900b7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6900ce550 0x7fb6900b4140 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 --2- 172.21.15.28:0/3345035195 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb69009cb20 0x7fb690158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 >> 172.21.15.28:0/3345035195 conn(0x7fb6900b54b0 msgr2=0x7fb690162e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 shutdown_connections 2024-04-25T19:12:38.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.059+0000 7fb695cb1640 1 -- 172.21.15.28:0/3345035195 wait complete. 2024-04-25T19:12:38.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-25T19:12:38.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.197+0000 7fb672861640 1 Processor -- start 2024-04-25T19:12:38.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.197+0000 7fb672861640 1 -- start start 2024-04-25T19:12:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb672861640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb66c0d22b0 0x7fb66c0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb672861640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb66c0b7880 0x7fb66c0b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb672861640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb66c09d260 con 0x7fb66c0d1ea0 2024-04-25T19:12:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb672861640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb66c09caf0 con 0x7fb66c0d22b0 2024-04-25T19:12:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb672861640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fb66c0d2890 con 0x7fb66c0b7880 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb66b7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb66c0d22b0 0x7fb66c0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb66bfff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb66c0d1ea0 0x7fb66c0b7170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36914/0 (socket says 172.21.15.28:36914) 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb66bfff640 1 -- 172.21.15.28:0/30825576 learned_addr learned my addr 172.21.15.28:0/30825576 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb670dfe640 1 --2- 172.21.15.28:0/30825576 >> v2:172.21.15.53:3300/0 conn(0x7fb66c0b7880 0x7fb66c0b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb66b7fe640 1 -- 172.21.15.28:0/30825576 >> v2:172.21.15.53:3300/0 conn(0x7fb66c0b7880 msgr2=0x7fb66c0b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/30825576 >> v2:172.21.15.53:3300/0 conn(0x7fb66c0b7880 0x7fb66c0b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb66b7fe640 1 -- 172.21.15.28:0/30825576 >> v1:172.21.15.28:6789/0 conn(0x7fb66c0d1ea0 legacy=0x7fb66c0b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb66affd640 1 -- 172.21.15.28:0/30825576 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 181634012 0 0) 0x7fb66c09d260 con 0x7fb66c0d1ea0 2024-04-25T19:12:38.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.199+0000 7fb66b7fe640 1 -- 172.21.15.28:0/30825576 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c0b9000 con 0x7fb66c0d22b0 2024-04-25T19:12:38.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.200+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/30825576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c0cf640 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0d15e0 tx=0x7fb65c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1382c5e38f10f3ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:38.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.200+0000 7fb66affd640 1 -- 172.21.15.28:0/30825576 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c037070 con 0x7fb66c0d22b0 2024-04-25T19:12:38.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.200+0000 7fb66affd640 1 -- 172.21.15.28:0/30825576 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb65c032440 con 0x7fb66c0d22b0 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.200+0000 7fb66affd640 1 -- 172.21.15.28:0/30825576 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65c02ae60 con 0x7fb66c0d22b0 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.201+0000 7fb672861640 1 -- 172.21.15.28:0/30825576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 msgr2=0x7fb66c0cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.201+0000 7fb672861640 1 --2- 172.21.15.28:0/30825576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c0cf640 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0d15e0 tx=0x7fb65c02a110 comp rx=0 tx=0).stop 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.201+0000 7fb672861640 1 -- 172.21.15.28:0/30825576 shutdown_connections 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.201+0000 7fb672861640 1 --2- 172.21.15.28:0/30825576 >> v2:172.21.15.53:3300/0 conn(0x7fb66c0b7880 0x7fb66c0b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.201+0000 7fb672861640 1 --2- 172.21.15.28:0/30825576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c0cf640 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.201+0000 7fb672861640 1 -- 172.21.15.28:0/30825576 >> 172.21.15.28:0/30825576 conn(0x7fb66c0b5400 msgr2=0x7fb66c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.201+0000 7fb672861640 1 -- 172.21.15.28:0/30825576 shutdown_connections 2024-04-25T19:12:38.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.201+0000 7fb672861640 1 -- 172.21.15.28:0/30825576 wait complete. 2024-04-25T19:12:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.203+0000 7fb672861640 1 Processor -- start 2024-04-25T19:12:38.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.203+0000 7fb672861640 1 -- start start 2024-04-25T19:12:38.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb672861640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0b7880 0x7fb66c0b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb672861640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb66c0d1ea0 0x7fb66c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb672861640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c1e0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb672861640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c0d1770 con 0x7fb66c0d22b0 2024-04-25T19:12:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb672861640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c0b81d0 con 0x7fb66c0b7880 2024-04-25T19:12:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb672861640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb66c0d18f0 con 0x7fb66c0d1ea0 2024-04-25T19:12:38.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb670dfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c1e0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb670dfe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c1e0f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49486/0 (socket says 172.21.15.28:49486) 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.204+0000 7fb670dfe640 1 -- 172.21.15.28:0/3713588205 learned_addr learned my addr 172.21.15.28:0/3713588205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb66bfff640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0b7880 0x7fb66c0b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb66c0d1ea0 0x7fb66c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb670dfe640 1 -- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb66c0d1ea0 msgr2=0x7fb66c0c8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb670dfe640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb66c0d1ea0 0x7fb66c0c8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb670dfe640 1 -- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0b7880 msgr2=0x7fb66c0b0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb670dfe640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0b7880 0x7fb66c0b0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb670dfe640 1 -- 172.21.15.28:0/3713588205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb66c1e14b0 con 0x7fb66c0d22b0 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb66b7fe640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb66c0d1ea0 0x7fb66c0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb670dfe640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c1e0f70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb664009cb0 tx=0x7fb66400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb668ff9640 1 -- 172.21.15.28:0/3713588205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664018020 con 0x7fb66c0d22b0 2024-04-25T19:12:38.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb668ff9640 1 -- 172.21.15.28:0/3713588205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb664014480 con 0x7fb66c0d22b0 2024-04-25T19:12:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb66c0c93f0 con 0x7fb66c0d22b0 2024-04-25T19:12:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb668ff9640 1 -- 172.21.15.28:0/3713588205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb664014870 con 0x7fb66c0d22b0 2024-04-25T19:12:38.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.205+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb66c0c99b0 con 0x7fb66c0d22b0 2024-04-25T19:12:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.206+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb66c050740 con 0x7fb66c0d22b0 2024-04-25T19:12:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.207+0000 7fb668ff9640 1 -- 172.21.15.28:0/3713588205 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb664005d10 con 0x7fb66c0d22b0 2024-04-25T19:12:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.207+0000 7fb668ff9640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6580744b0 0x7fb6580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.207+0000 7fb668ff9640 1 -- 172.21.15.28:0/3713588205 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fb664095e90 con 0x7fb66c0d22b0 2024-04-25T19:12:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.208+0000 7fb66bfff640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6580744b0 0x7fb6580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.208+0000 7fb66bfff640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6580744b0 0x7fb6580769c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb660000f60 tx=0x7fb660007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:38.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.212+0000 7fb668ff9640 1 -- 172.21.15.28:0/3713588205 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb664022030 con 0x7fb66c0d22b0 2024-04-25T19:12:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.471+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7fb66c0c9180 con 0x7fb66c0d22b0 2024-04-25T19:12:38.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.473+0000 7fb668ff9640 1 -- 172.21.15.28:0/3713588205 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7fb66c1e14b0 con 0x7fb66c0d22b0 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.475+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6580744b0 msgr2=0x7fb6580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.475+0000 7fb672861640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6580744b0 0x7fb6580769c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb660000f60 tx=0x7fb660007c00 comp rx=0 tx=0).stop 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.475+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 msgr2=0x7fb66c1e0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.475+0000 7fb672861640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c1e0f70 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb664009cb0 tx=0x7fb66400c490 comp rx=0 tx=0).stop 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.476+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 shutdown_connections 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.476+0000 7fb672861640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6580744b0 0x7fb6580769c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.476+0000 7fb672861640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb66c0d22b0 0x7fb66c1e0f70 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.476+0000 7fb672861640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb66c0d1ea0 0x7fb66c0c8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.476+0000 7fb672861640 1 --2- 172.21.15.28:0/3713588205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0b7880 0x7fb66c0b0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.476+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 >> 172.21.15.28:0/3713588205 conn(0x7fb66c0b5400 msgr2=0x7fb66c071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.476+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 shutdown_connections 2024-04-25T19:12:38.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.476+0000 7fb672861640 1 -- 172.21.15.28:0/3713588205 wait complete. 2024-04-25T19:12:38.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-25T19:12:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.600+0000 7f46abe6e640 1 Processor -- start 2024-04-25T19:12:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.600+0000 7f46abe6e640 1 -- start start 2024-04-25T19:12:38.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46abe6e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a4160310 0x7f46a40d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46abe6e640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f46a40b79f0 0x7f46a40b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46abe6e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46a4160720 con 0x7f46a408cce0 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46abe6e640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f46a40b8340 con 0x7f46a40b79f0 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46abe6e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f46a40d5640 con 0x7f46a4160310 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46a9c0a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a4160310 0x7f46a40d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46a9409640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f46a40b79f0 0x7f46a40b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46aa40b640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f46a408cce0 0x7f46a408d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:37436/0 (socket says 172.21.15.28:37436) 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46a9c0a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a4160310 0x7f46a40d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:44788/0 (socket says 172.21.15.28:44788) 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46aa40b640 1 -- 172.21.15.28:0/2340455781 learned_addr learned my addr 172.21.15.28:0/2340455781 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46a9409640 1 -- 172.21.15.28:0/2340455781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a4160310 msgr2=0x7f46a40d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46a9409640 1 --2- 172.21.15.28:0/2340455781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a4160310 0x7f46a40d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46a9409640 1 -- 172.21.15.28:0/2340455781 >> v1:172.21.15.53:6789/0 conn(0x7f46a408cce0 legacy=0x7f46a408d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46a9409640 1 -- 172.21.15.28:0/2340455781 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a40d0f40 con 0x7f46a40b79f0 2024-04-25T19:12:38.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.602+0000 7f46a9c0a640 1 --2- 172.21.15.28:0/2340455781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a4160310 0x7f46a40d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:38.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.603+0000 7f46a9409640 1 --2- 172.21.15.28:0/2340455781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 0x7f46a40b7e00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46980097a0 tx=0x7f469802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ee9d39a96dfdd2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:38.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.603+0000 7f46a8c08640 1 -- 172.21.15.28:0/2340455781 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698037070 con 0x7f46a40b79f0 2024-04-25T19:12:38.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.603+0000 7f46a8c08640 1 -- 172.21.15.28:0/2340455781 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4698032440 con 0x7f46a40b79f0 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46a8c08640 1 -- 172.21.15.28:0/2340455781 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4698032810 con 0x7f46a40b79f0 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46abe6e640 1 -- 172.21.15.28:0/2340455781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 msgr2=0x7f46a40b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2340455781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 0x7f46a40b7e00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46980097a0 tx=0x7f469802a180 comp rx=0 tx=0).stop 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46abe6e640 1 -- 172.21.15.28:0/2340455781 shutdown_connections 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2340455781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 0x7f46a40b7e00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2340455781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a4160310 0x7f46a40d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46abe6e640 1 -- 172.21.15.28:0/2340455781 >> 172.21.15.28:0/2340455781 conn(0x7f46a40d2a50 msgr2=0x7f46a40d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46abe6e640 1 -- 172.21.15.28:0/2340455781 shutdown_connections 2024-04-25T19:12:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.604+0000 7f46abe6e640 1 -- 172.21.15.28:0/2340455781 wait complete. 2024-04-25T19:12:38.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.606+0000 7f46abe6e640 1 Processor -- start 2024-04-25T19:12:38.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.606+0000 7f46abe6e640 1 -- start start 2024-04-25T19:12:38.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.607+0000 7f46abe6e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a408cce0 0x7f46a40ce160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.607+0000 7f46abe6e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 0x7f46a415b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.607+0000 7f46abe6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a4160310 0x7f46a41e6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.607+0000 7f46abe6e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f46a40b9400 con 0x7f46a4160310 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.607+0000 7f46abe6e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f46a40d1500 con 0x7f46a40b79f0 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.607+0000 7f46abe6e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f46a40d0bf0 con 0x7f46a408cce0 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.607+0000 7f46aa40b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a4160310 0x7f46a41e6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46aa40b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a4160310 0x7f46a41e6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49502/0 (socket says 172.21.15.28:49502) 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46aa40b640 1 -- 172.21.15.28:0/2521305975 learned_addr learned my addr 172.21.15.28:0/2521305975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46a9409640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 0x7f46a415b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46a9c0a640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a408cce0 0x7f46a40ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46aa40b640 1 -- 172.21.15.28:0/2521305975 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a408cce0 msgr2=0x7f46a40ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46aa40b640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a408cce0 0x7f46a40ce160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46aa40b640 1 -- 172.21.15.28:0/2521305975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 msgr2=0x7f46a415b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46aa40b640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 0x7f46a415b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46aa40b640 1 -- 172.21.15.28:0/2521305975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46a415ba30 con 0x7f46a4160310 2024-04-25T19:12:38.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46aa40b640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a4160310 0x7f46a41e6a90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f469400cad0 tx=0x7f469400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:38.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46a27fc640 1 -- 172.21.15.28:0/2521305975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694017070 con 0x7f46a4160310 2024-04-25T19:12:38.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46a27fc640 1 -- 172.21.15.28:0/2521305975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f469400ac30 con 0x7f46a4160310 2024-04-25T19:12:38.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46a40cd0c0 con 0x7f46a4160310 2024-04-25T19:12:38.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46a27fc640 1 -- 172.21.15.28:0/2521305975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4694012400 con 0x7f46a4160310 2024-04-25T19:12:38.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.608+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46a40cd680 con 0x7f46a4160310 2024-04-25T19:12:38.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.609+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46a40d1da0 con 0x7f46a4160310 2024-04-25T19:12:38.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.610+0000 7f46a27fc640 1 -- 172.21.15.28:0/2521305975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f469401a070 con 0x7f46a4160310 2024-04-25T19:12:38.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.610+0000 7f46a27fc640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f46900744b0 0x7f46900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:38.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.610+0000 7f46a27fc640 1 -- 172.21.15.28:0/2521305975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f4694094ce0 con 0x7f46a4160310 2024-04-25T19:12:38.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.610+0000 7f46a9c0a640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f46900744b0 0x7f46900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:38.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.611+0000 7f46a9c0a640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f46900744b0 0x7f46900769c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f469c009560 tx=0x7f469c009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:38.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.615+0000 7f46a27fc640 1 -- 172.21.15.28:0/2521305975 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f46a40d1da0 con 0x7f46a4160310 2024-04-25T19:12:38.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.875+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f46a40b7e00 con 0x7f46a4160310 2024-04-25T19:12:38.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.878+0000 7f46a27fc640 1 -- 172.21.15.28:0/2521305975 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f469409b080 con 0x7f46a4160310 2024-04-25T19:12:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f46900744b0 msgr2=0x7f46900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f46900744b0 0x7f46900769c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f469c009560 tx=0x7f469c009190 comp rx=0 tx=0).stop 2024-04-25T19:12:38.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a4160310 msgr2=0x7f46a41e6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a4160310 0x7f46a41e6a90 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f469400cad0 tx=0x7f469400cf90 comp rx=0 tx=0).stop 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 shutdown_connections 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f46900744b0 0x7f46900769c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a4160310 0x7f46a41e6a90 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46a40b79f0 0x7f46a415b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 --2- 172.21.15.28:0/2521305975 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f46a408cce0 0x7f46a40ce160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 >> 172.21.15.28:0/2521305975 conn(0x7f46a40d2a50 msgr2=0x7f46a40bac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 shutdown_connections 2024-04-25T19:12:38.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:38.880+0000 7f46abe6e640 1 -- 172.21.15.28:0/2521305975 wait complete. 2024-04-25T19:12:38.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-25T19:12:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.004+0000 7f53068ee640 1 Processor -- start 2024-04-25T19:12:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.004+0000 7f53068ee640 1 -- start start 2024-04-25T19:12:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f53068ee640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53000b9000 0x7f53000d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f53068ee640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f530006e8b0 0x7f53000cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f53068ee640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53000b9830 con 0x7f530006e4a0 2024-04-25T19:12:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f53068ee640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f53000d2ea0 con 0x7f53000b9000 2024-04-25T19:12:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f53068ee640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f53000d0ec0 con 0x7f530006e8b0 2024-04-25T19:12:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f52fffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53000b9000 0x7f53000d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f52fffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53000b9000 0x7f53000d3310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49504/0 (socket says 172.21.15.28:49504) 2024-04-25T19:12:39.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f52fffff640 1 -- 172.21.15.28:0/1085751267 learned_addr learned my addr 172.21.15.28:0/1085751267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f5304e8b640 1 --2- 172.21.15.28:0/1085751267 >> v2:172.21.15.53:3300/0 conn(0x7f530006e8b0 0x7f53000cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f52fffff640 1 -- 172.21.15.28:0/1085751267 >> v2:172.21.15.53:3300/0 conn(0x7f530006e8b0 msgr2=0x7f53000cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f52fffff640 1 --2- 172.21.15.28:0/1085751267 >> v2:172.21.15.53:3300/0 conn(0x7f530006e8b0 0x7f53000cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f52fffff640 1 -- 172.21.15.28:0/1085751267 >> v1:172.21.15.53:6789/0 conn(0x7f530006e4a0 legacy=0x7f53000ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.006+0000 7f52fffff640 1 -- 172.21.15.28:0/1085751267 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530009d230 con 0x7f53000b9000 2024-04-25T19:12:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.007+0000 7f52fffff640 1 --2- 172.21.15.28:0/1085751267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53000b9000 0x7f53000d3310 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5300073ea0 tx=0x7f52f402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e31c3e1f1a10f04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.007+0000 7f52feffd640 1 -- 172.21.15.28:0/1085751267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4037070 con 0x7f53000b9000 2024-04-25T19:12:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.007+0000 7f52feffd640 1 -- 172.21.15.28:0/1085751267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f4032440 con 0x7f53000b9000 2024-04-25T19:12:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.007+0000 7f52feffd640 1 -- 172.21.15.28:0/1085751267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4032810 con 0x7f53000b9000 2024-04-25T19:12:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.008+0000 7f53068ee640 1 -- 172.21.15.28:0/1085751267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53000b9000 msgr2=0x7f53000d3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.008+0000 7f53068ee640 1 --2- 172.21.15.28:0/1085751267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53000b9000 0x7f53000d3310 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5300073ea0 tx=0x7f52f402a270 comp rx=0 tx=0).stop 2024-04-25T19:12:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.008+0000 7f53068ee640 1 -- 172.21.15.28:0/1085751267 shutdown_connections 2024-04-25T19:12:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.008+0000 7f53068ee640 1 --2- 172.21.15.28:0/1085751267 >> v2:172.21.15.53:3300/0 conn(0x7f530006e8b0 0x7f53000cf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.008+0000 7f53068ee640 1 --2- 172.21.15.28:0/1085751267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53000b9000 0x7f53000d3310 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.008+0000 7f53068ee640 1 -- 172.21.15.28:0/1085751267 >> 172.21.15.28:0/1085751267 conn(0x7f53000d03a0 msgr2=0x7f53000d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.008+0000 7f53068ee640 1 -- 172.21.15.28:0/1085751267 shutdown_connections 2024-04-25T19:12:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.008+0000 7f53068ee640 1 -- 172.21.15.28:0/1085751267 wait complete. 2024-04-25T19:12:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.010+0000 7f53068ee640 1 Processor -- start 2024-04-25T19:12:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.010+0000 7f53068ee640 1 -- start start 2024-04-25T19:12:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f53068ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530006e4a0 0x7f530015fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f53068ee640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f530006e8b0 0x7f530014fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f53068ee640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53000b9000 0x7f530015b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f53068ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f530009cf50 con 0x7f530006e4a0 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f53068ee640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f530009cd00 con 0x7f53000b9000 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f53068ee640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f53000cf5d0 con 0x7f530006e8b0 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f52fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530006e4a0 0x7f530015fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f52fffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530006e4a0 0x7f530015fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49514/0 (socket says 172.21.15.28:49514) 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f52fffff640 1 -- 172.21.15.28:0/219754922 learned_addr learned my addr 172.21.15.28:0/219754922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f5304e8b640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53000b9000 0x7f530015b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.011+0000 7f52ff7fe640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f530006e8b0 0x7f530014fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fffff640 1 -- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f530006e8b0 msgr2=0x7f530014fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fffff640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f530006e8b0 0x7f530014fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fffff640 1 -- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53000b9000 msgr2=0x7f530015b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fffff640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53000b9000 0x7f530015b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fffff640 1 -- 172.21.15.28:0/219754922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f530015d910 con 0x7f530006e4a0 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f5304e8b640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53000b9000 0x7f530015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fffff640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530006e4a0 0x7f530015fac0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f52f402a2b0 tx=0x7f52f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fcff9640 1 -- 172.21.15.28:0/219754922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4041020 con 0x7f530006e4a0 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fcff9640 1 -- 172.21.15.28:0/219754922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f40306c0 con 0x7f530006e4a0 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f52fcff9640 1 -- 172.21.15.28:0/219754922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f4037040 con 0x7f530006e4a0 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f530015dbf0 con 0x7f530006e4a0 2024-04-25T19:12:39.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.012+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f530015e1b0 con 0x7f530006e4a0 2024-04-25T19:12:39.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.013+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5300050740 con 0x7f530006e4a0 2024-04-25T19:12:39.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.014+0000 7f52fcff9640 1 -- 172.21.15.28:0/219754922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f52f4044020 con 0x7f530006e4a0 2024-04-25T19:12:39.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.014+0000 7f52fcff9640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ec0744b0 0x7f52ec0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.014+0000 7f52fcff9640 1 -- 172.21.15.28:0/219754922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f52f40b5800 con 0x7f530006e4a0 2024-04-25T19:12:39.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.014+0000 7f52ff7fe640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ec0744b0 0x7f52ec0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.015+0000 7f52ff7fe640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ec0744b0 0x7f52ec0769c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f530015aed0 tx=0x7f52f0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.019+0000 7f52fcff9640 1 -- 172.21.15.28:0/219754922 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52f404b030 con 0x7f530006e4a0 2024-04-25T19:12:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.285+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f53000715a0 con 0x7f530006e4a0 2024-04-25T19:12:39.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.287+0000 7f52fcff9640 1 -- 172.21.15.28:0/219754922 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f52f40801e0 con 0x7f530006e4a0 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ec0744b0 msgr2=0x7f52ec0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ec0744b0 0x7f52ec0769c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f530015aed0 tx=0x7f52f0009040 comp rx=0 tx=0).stop 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530006e4a0 msgr2=0x7f530015fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530006e4a0 0x7f530015fac0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f52f402a2b0 tx=0x7f52f40046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 shutdown_connections 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ec0744b0 0x7f52ec0769c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53000b9000 0x7f530015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f530006e8b0 0x7f530014fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 --2- 172.21.15.28:0/219754922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f530006e4a0 0x7f530015fac0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.289+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 >> 172.21.15.28:0/219754922 conn(0x7f53000d03a0 msgr2=0x7f5300163e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.290+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 shutdown_connections 2024-04-25T19:12:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.290+0000 7f53068ee640 1 -- 172.21.15.28:0/219754922 wait complete. 2024-04-25T19:12:39.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-25T19:12:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.414+0000 7f9146e27640 1 Processor -- start 2024-04-25T19:12:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.414+0000 7f9146e27640 1 -- start start 2024-04-25T19:12:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f9146e27640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9140164990 0x7f914015dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f9146e27640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f9140160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f9146e27640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91401614a0 con 0x7f91400b7660 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f9146e27640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f9140163eb0 con 0x7f9140164990 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f9146e27640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9140164190 con 0x7f91401606f0 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f913ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f91400b7660 0x7f91400b7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36920/0 (socket says 172.21.15.28:36920) 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f9144bc3640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9140164990 0x7f914015dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f913ffff640 1 -- 172.21.15.28:0/2576838323 learned_addr learned my addr 172.21.15.28:0/2576838323 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.415+0000 7f91453c4640 1 --2- 172.21.15.28:0/2576838323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f9140160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.416+0000 7f9144bc3640 1 -- 172.21.15.28:0/2576838323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 msgr2=0x7f9140160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.416+0000 7f9144bc3640 1 --2- 172.21.15.28:0/2576838323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f9140160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.416+0000 7f9144bc3640 1 -- 172.21.15.28:0/2576838323 >> v1:172.21.15.28:6789/0 conn(0x7f91400b7660 legacy=0x7f91400b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.416+0000 7f913f7fe640 1 -- 172.21.15.28:0/2576838323 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3307209419 0 0) 0x7f91401614a0 con 0x7f91400b7660 2024-04-25T19:12:39.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.416+0000 7f9144bc3640 1 -- 172.21.15.28:0/2576838323 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9140164470 con 0x7f9140164990 2024-04-25T19:12:39.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.416+0000 7f91453c4640 1 --2- 172.21.15.28:0/2576838323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f9140160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.416+0000 7f9144bc3640 1 --2- 172.21.15.28:0/2576838323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f914015dcb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f91300097a0 tx=0x7f913002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ffd7c2bd60f4d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f913f7fe640 1 -- 172.21.15.28:0/2576838323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9130037070 con 0x7f9140164990 2024-04-25T19:12:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f913f7fe640 1 -- 172.21.15.28:0/2576838323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9130032440 con 0x7f9140164990 2024-04-25T19:12:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f913f7fe640 1 -- 172.21.15.28:0/2576838323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9130032810 con 0x7f9140164990 2024-04-25T19:12:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f9146e27640 1 -- 172.21.15.28:0/2576838323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 msgr2=0x7f914015dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f9146e27640 1 --2- 172.21.15.28:0/2576838323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f914015dcb0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f91300097a0 tx=0x7f913002a180 comp rx=0 tx=0).stop 2024-04-25T19:12:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f9146e27640 1 -- 172.21.15.28:0/2576838323 shutdown_connections 2024-04-25T19:12:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f9146e27640 1 --2- 172.21.15.28:0/2576838323 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f9140160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f9146e27640 1 --2- 172.21.15.28:0/2576838323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f914015dcb0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f9146e27640 1 -- 172.21.15.28:0/2576838323 >> 172.21.15.28:0/2576838323 conn(0x7f91400d2840 msgr2=0x7f91400d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f9146e27640 1 -- 172.21.15.28:0/2576838323 shutdown_connections 2024-04-25T19:12:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.417+0000 7f9146e27640 1 -- 172.21.15.28:0/2576838323 wait complete. 2024-04-25T19:12:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.419+0000 7f9146e27640 1 Processor -- start 2024-04-25T19:12:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.419+0000 7f9146e27640 1 -- start start 2024-04-25T19:12:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.420+0000 7f9146e27640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91400b7660 0x7f91401503b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f9146e27640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f91401e9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f9146e27640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f91401ea180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f9146e27640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f91400b45d0 con 0x7f9140164990 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f9146e27640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9140161250 con 0x7f91400b7660 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f9146e27640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f91400b4750 con 0x7f91401606f0 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f91401ea180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f91401ea180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49542/0 (socket says 172.21.15.28:49542) 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 -- 172.21.15.28:0/3686047428 learned_addr learned my addr 172.21.15.28:0/3686047428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f9144bc3640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91400b7660 0x7f91401503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f913ffff640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f91401e9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 -- 172.21.15.28:0/3686047428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 msgr2=0x7f91401e9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f91401e9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 -- 172.21.15.28:0/3686047428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91400b7660 msgr2=0x7f91401503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91400b7660 0x7f91401503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 -- 172.21.15.28:0/3686047428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91401ea9c0 con 0x7f9140164990 2024-04-25T19:12:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f91453c4640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f91401ea180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9138007dc0 tx=0x7f913800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f913d7fa640 1 -- 172.21.15.28:0/3686047428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913801a070 con 0x7f9140164990 2024-04-25T19:12:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f913d7fa640 1 -- 172.21.15.28:0/3686047428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9138003990 con 0x7f9140164990 2024-04-25T19:12:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91401eaca0 con 0x7f9140164990 2024-04-25T19:12:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f913d7fa640 1 -- 172.21.15.28:0/3686047428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9138003d80 con 0x7f9140164990 2024-04-25T19:12:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.421+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91401eb260 con 0x7f9140164990 2024-04-25T19:12:39.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.422+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9140050740 con 0x7f9140164990 2024-04-25T19:12:39.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.423+0000 7f913d7fa640 1 -- 172.21.15.28:0/3686047428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f913801d070 con 0x7f9140164990 2024-04-25T19:12:39.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.424+0000 7f913d7fa640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f912c0744e0 0x7f912c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.424+0000 7f913d7fa640 1 -- 172.21.15.28:0/3686047428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f9138095210 con 0x7f9140164990 2024-04-25T19:12:39.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.424+0000 7f9144bc3640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f912c0744e0 0x7f912c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.424+0000 7f9144bc3640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f912c0744e0 0x7f912c0769f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9130034000 tx=0x7f913002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.428+0000 7f913d7fa640 1 -- 172.21.15.28:0/3686047428 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9138010070 con 0x7f9140164990 2024-04-25T19:12:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.685+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f91401ea750 con 0x7f9140164990 2024-04-25T19:12:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.688+0000 7f913d7fa640 1 -- 172.21.15.28:0/3686047428 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f913805f500 con 0x7f9140164990 2024-04-25T19:12:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f912c0744e0 msgr2=0x7f912c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f912c0744e0 0x7f912c0769f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9130034000 tx=0x7f913002e000 comp rx=0 tx=0).stop 2024-04-25T19:12:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 msgr2=0x7f91401ea180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f91401ea180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9138007dc0 tx=0x7f913800d4a0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 shutdown_connections 2024-04-25T19:12:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f912c0744e0 0x7f912c0769f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9140164990 0x7f91401ea180 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91401606f0 0x7f91401e9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 --2- 172.21.15.28:0/3686047428 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91400b7660 0x7f91401503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 >> 172.21.15.28:0/3686047428 conn(0x7f91400d2840 msgr2=0x7f91400b6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 shutdown_connections 2024-04-25T19:12:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.690+0000 7f9146e27640 1 -- 172.21.15.28:0/3686047428 wait complete. 2024-04-25T19:12:39.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-25T19:12:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.816+0000 7fe5d699f640 1 Processor -- start 2024-04-25T19:12:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.816+0000 7fe5d699f640 1 -- start start 2024-04-25T19:12:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.817+0000 7fe5d699f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5d00b1f60 0x7fe5d00b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.817+0000 7fe5d699f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00d0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.817+0000 7fe5d699f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5d00d0aa0 con 0x7fe5d00d0690 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.817+0000 7fe5d699f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fe5d00d2be0 con 0x7fe5d00b1f60 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.817+0000 7fe5d699f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe5d00cfa80 con 0x7fe5d00cefc0 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.817+0000 7fe5d599d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5d00b1f60 0x7fe5d00b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.817+0000 7fe5d599d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe5d00b1f60 0x7fe5d00b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49550/0 (socket says 172.21.15.28:49550) 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.817+0000 7fe5d599d640 1 -- 172.21.15.28:0/1673009405 learned_addr learned my addr 172.21.15.28:0/1673009405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.818+0000 7fe5d519c640 1 --2- 172.21.15.28:0/1673009405 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.818+0000 7fe5d599d640 1 -- 172.21.15.28:0/1673009405 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 msgr2=0x7fe5d00d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.818+0000 7fe5d599d640 1 --2- 172.21.15.28:0/1673009405 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00d0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.818+0000 7fe5d599d640 1 -- 172.21.15.28:0/1673009405 >> v1:172.21.15.53:6789/0 conn(0x7fe5d00d0690 legacy=0x7fe5d0161b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.818+0000 7fe5d599d640 1 -- 172.21.15.28:0/1673009405 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d00cf4c0 con 0x7fe5d00b1f60 2024-04-25T19:12:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.818+0000 7fe5d519c640 1 --2- 172.21.15.28:0/1673009405 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.818+0000 7fe5d599d640 1 --2- 172.21.15.28:0/1673009405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00b1f60 0x7fe5d00b28a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009870 tx=0x7fe5c802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc64f7ed401153a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.818+0000 7fe5d499b640 1 -- 172.21.15.28:0/1673009405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8037070 con 0x7fe5d00b1f60 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d499b640 1 -- 172.21.15.28:0/1673009405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8032440 con 0x7fe5d00b1f60 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d499b640 1 -- 172.21.15.28:0/1673009405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8032810 con 0x7fe5d00b1f60 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d699f640 1 -- 172.21.15.28:0/1673009405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00b1f60 msgr2=0x7fe5d00b28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d699f640 1 --2- 172.21.15.28:0/1673009405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00b1f60 0x7fe5d00b28a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009870 tx=0x7fe5c802a180 comp rx=0 tx=0).stop 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d699f640 1 -- 172.21.15.28:0/1673009405 shutdown_connections 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d699f640 1 --2- 172.21.15.28:0/1673009405 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00d0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d699f640 1 --2- 172.21.15.28:0/1673009405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00b1f60 0x7fe5d00b28a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d699f640 1 -- 172.21.15.28:0/1673009405 >> 172.21.15.28:0/1673009405 conn(0x7fe5d00d3760 msgr2=0x7fe5d00d3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d699f640 1 -- 172.21.15.28:0/1673009405 shutdown_connections 2024-04-25T19:12:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.819+0000 7fe5d699f640 1 -- 172.21.15.28:0/1673009405 wait complete. 2024-04-25T19:12:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.821+0000 7fe5d699f640 1 Processor -- start 2024-04-25T19:12:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.821+0000 7fe5d699f640 1 -- start start 2024-04-25T19:12:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.822+0000 7fe5d699f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5d00b1f60 0x7fe5d01f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.822+0000 7fe5d699f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.822+0000 7fe5d699f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00d0690 0x7fe5d0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.822+0000 7fe5d699f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d00d16b0 con 0x7fe5d00d0690 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.822+0000 7fe5d699f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d00d13d0 con 0x7fe5d00b1f60 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.822+0000 7fe5d699f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe5d00b2de0 con 0x7fe5d00cefc0 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00d0690 0x7fe5d0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00d0690 0x7fe5d0071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49552/0 (socket says 172.21.15.28:49552) 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 -- 172.21.15.28:0/4175336596 learned_addr learned my addr 172.21.15.28:0/4175336596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d519c640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d599d640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5d00b1f60 0x7fe5d01f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 -- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 msgr2=0x7fe5d00a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 -- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5d00b1f60 msgr2=0x7fe5d01f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5d00b1f60 0x7fe5d01f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 -- 172.21.15.28:0/4175336596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d01f78d0 con 0x7fe5d00d0690 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d599d640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5d00b1f60 0x7fe5d01f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d619e640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00d0690 0x7fe5d0071e60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000ba10 tx=0x7fe5c000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5ce7fc640 1 -- 172.21.15.28:0/4175336596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0019070 con 0x7fe5d00d0690 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5ce7fc640 1 -- 172.21.15.28:0/4175336596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c000cb40 con 0x7fe5d00d0690 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5d01f7bb0 con 0x7fe5d00d0690 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5ce7fc640 1 -- 172.21.15.28:0/4175336596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0014440 con 0x7fe5d00d0690 2024-04-25T19:12:39.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.823+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5d009e4e0 con 0x7fe5d00d0690 2024-04-25T19:12:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.824+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5d0050740 con 0x7fe5d00d0690 2024-04-25T19:12:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.825+0000 7fe5ce7fc640 1 -- 172.21.15.28:0/4175336596 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe5c001c070 con 0x7fe5d00d0690 2024-04-25T19:12:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.825+0000 7fe5ce7fc640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe5bc0744b0 0x7fe5bc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.826+0000 7fe5ce7fc640 1 -- 172.21.15.28:0/4175336596 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fe5c0094d50 con 0x7fe5d00d0690 2024-04-25T19:12:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.826+0000 7fe5d599d640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe5bc0744b0 0x7fe5bc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:39.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.826+0000 7fe5d599d640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe5bc0744b0 0x7fe5bc0769c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8034000 tx=0x7fe5c802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:39.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:39.830+0000 7fe5ce7fc640 1 -- 172.21.15.28:0/4175336596 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe5c0098030 con 0x7fe5d00d0690 2024-04-25T19:12:40.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.086+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fe5d00a1250 con 0x7fe5d00d0690 2024-04-25T19:12:40.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.089+0000 7fe5ce7fc640 1 -- 172.21.15.28:0/4175336596 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7fe5c005f6a0 con 0x7fe5d00d0690 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe5bc0744b0 msgr2=0x7fe5bc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe5bc0744b0 0x7fe5bc0769c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8034000 tx=0x7fe5c802e000 comp rx=0 tx=0).stop 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00d0690 msgr2=0x7fe5d0071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00d0690 0x7fe5d0071e60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fe5c000ba10 tx=0x7fe5c000bed0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 shutdown_connections 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe5bc0744b0 0x7fe5bc0769c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5d00d0690 0x7fe5d0071e60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe5d00cefc0 0x7fe5d00a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 --2- 172.21.15.28:0/4175336596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5d00b1f60 0x7fe5d01f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 >> 172.21.15.28:0/4175336596 conn(0x7fe5d00d3760 msgr2=0x7fe5d00b6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 shutdown_connections 2024-04-25T19:12:40.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.091+0000 7fe5d699f640 1 -- 172.21.15.28:0/4175336596 wait complete. 2024-04-25T19:12:40.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-25T19:12:40.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.214+0000 7f8fbc54d640 1 Processor -- start 2024-04-25T19:12:40.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.214+0000 7f8fbc54d640 1 -- start start 2024-04-25T19:12:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fbc54d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fb4160250 0x7f8fb40b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fbc54d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8fb40d2ca0 0x7f8fb40d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fbc54d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fb4164b20 con 0x7f8fb40b81a0 2024-04-25T19:12:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fbc54d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8fb40b4f50 con 0x7f8fb4160250 2024-04-25T19:12:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fbc54d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f8fb4163d60 con 0x7f8fb40d2ca0 2024-04-25T19:12:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fba2e9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fb4160250 0x7f8fb40b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fba2e9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8fb4160250 0x7f8fb40b4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49564/0 (socket says 172.21.15.28:49564) 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fba2e9640 1 -- 172.21.15.28:0/326892760 learned_addr learned my addr 172.21.15.28:0/326892760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fbaaea640 1 --2- 172.21.15.28:0/326892760 >> v2:172.21.15.53:3300/0 conn(0x7f8fb40d2ca0 0x7f8fb40d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fba2e9640 1 -- 172.21.15.28:0/326892760 >> v2:172.21.15.53:3300/0 conn(0x7f8fb40d2ca0 msgr2=0x7f8fb40d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fba2e9640 1 --2- 172.21.15.28:0/326892760 >> v2:172.21.15.53:3300/0 conn(0x7f8fb40d2ca0 0x7f8fb40d44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fba2e9640 1 -- 172.21.15.28:0/326892760 >> v1:172.21.15.53:6789/0 conn(0x7f8fb40b81a0 legacy=0x7f8fb40d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fba2e9640 1 -- 172.21.15.28:0/326892760 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb4164040 con 0x7f8fb4160250 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.216+0000 7f8fbaaea640 1 --2- 172.21.15.28:0/326892760 >> v2:172.21.15.53:3300/0 conn(0x7f8fb40d2ca0 0x7f8fb40d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.217+0000 7f8fba2e9640 1 --2- 172.21.15.28:0/326892760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb4160250 0x7f8fb40b4690 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40d3890 tx=0x7f8fac02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41179e01e6e345b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.217+0000 7f8fb92e7640 1 -- 172.21.15.28:0/326892760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac037070 con 0x7f8fb4160250 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.217+0000 7f8fb92e7640 1 -- 172.21.15.28:0/326892760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fac032440 con 0x7f8fb4160250 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.217+0000 7f8fb92e7640 1 -- 172.21.15.28:0/326892760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fac032810 con 0x7f8fb4160250 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.218+0000 7f8fbc54d640 1 -- 172.21.15.28:0/326892760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb4160250 msgr2=0x7f8fb40b4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.218+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/326892760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb4160250 0x7f8fb40b4690 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40d3890 tx=0x7f8fac02a1a0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.218+0000 7f8fbc54d640 1 -- 172.21.15.28:0/326892760 shutdown_connections 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.218+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/326892760 >> v2:172.21.15.53:3300/0 conn(0x7f8fb40d2ca0 0x7f8fb40d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.218+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/326892760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb4160250 0x7f8fb40b4690 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.218+0000 7f8fbc54d640 1 -- 172.21.15.28:0/326892760 >> 172.21.15.28:0/326892760 conn(0x7f8fb406ee30 msgr2=0x7f8fb406c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.218+0000 7f8fbc54d640 1 -- 172.21.15.28:0/326892760 shutdown_connections 2024-04-25T19:12:40.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.218+0000 7f8fbc54d640 1 -- 172.21.15.28:0/326892760 wait complete. 2024-04-25T19:12:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.220+0000 7f8fbc54d640 1 Processor -- start 2024-04-25T19:12:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.220+0000 7f8fbc54d640 1 -- start start 2024-04-25T19:12:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbc54d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8fb40b81a0 0x7f8fb40cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbc54d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fb4160250 0x7f8fb4160f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbc54d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb40cc100 0x7f8fb415ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbc54d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb40b4bd0 con 0x7f8fb40cc100 2024-04-25T19:12:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbc54d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8fb4164600 con 0x7f8fb4160250 2024-04-25T19:12:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbc54d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8fb40d49f0 con 0x7f8fb40b81a0 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbaaea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb40cc100 0x7f8fb415ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbaaea640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb40cc100 0x7f8fb415ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49570/0 (socket says 172.21.15.28:49570) 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbaaea640 1 -- 172.21.15.28:0/1046722944 learned_addr learned my addr 172.21.15.28:0/1046722944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fba2e9640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8fb40b81a0 0x7f8fb40cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fb9ae8640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fb4160250 0x7f8fb4160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbaaea640 1 -- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8fb40b81a0 msgr2=0x7f8fb40cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbaaea640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8fb40b81a0 0x7f8fb40cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbaaea640 1 -- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fb4160250 msgr2=0x7f8fb4160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.221+0000 7f8fbaaea640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fb4160250 0x7f8fb4160f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.222+0000 7f8fbaaea640 1 -- 172.21.15.28:0/1046722944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fb4161c80 con 0x7f8fb40cc100 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.222+0000 7f8fba2e9640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8fb40b81a0 0x7f8fb40cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.222+0000 7f8fbaaea640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb40cc100 0x7f8fb415ac60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400cd80 tx=0x7f8fa400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.222+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/1046722944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa400f680 con 0x7f8fb40cc100 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.222+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/1046722944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fa400f840 con 0x7f8fb40cc100 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.222+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fb4161f60 con 0x7f8fb40cc100 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.222+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/1046722944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fa400fc30 con 0x7f8fb40cc100 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.222+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fb4162520 con 0x7f8fb40cc100 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.223+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fb41619f0 con 0x7f8fb40cc100 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.224+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/1046722944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8fa40213f0 con 0x7f8fb40cc100 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.224+0000 7f8fb2ffd640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8fa00744b0 0x7f8fa00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.224+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/1046722944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f8fa4095e90 con 0x7f8fb40cc100 2024-04-25T19:12:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.224+0000 7f8fba2e9640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8fa00744b0 0x7f8fa00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.225+0000 7f8fba2e9640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8fa00744b0 0x7f8fa00769c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8fac034000 tx=0x7f8fac02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.229+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/1046722944 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8fa4014030 con 0x7f8fb40cc100 2024-04-25T19:12:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.483+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f8fb4050740 con 0x7f8fb40cc100 2024-04-25T19:12:40.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.486+0000 7f8fb2ffd640 1 -- 172.21.15.28:0/1046722944 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f8fa40607e0 con 0x7f8fb40cc100 2024-04-25T19:12:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.488+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8fa00744b0 msgr2=0x7f8fa00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.488+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8fa00744b0 0x7f8fa00769c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8fac034000 tx=0x7f8fac02e000 comp rx=0 tx=0).stop 2024-04-25T19:12:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb40cc100 msgr2=0x7f8fb415ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb40cc100 0x7f8fb415ac60 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f8fa400cd80 tx=0x7f8fa400a650 comp rx=0 tx=0).stop 2024-04-25T19:12:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 shutdown_connections 2024-04-25T19:12:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8fa00744b0 0x7f8fa00769c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8fb40cc100 0x7f8fb415ac60 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8fb4160250 0x7f8fb4160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 --2- 172.21.15.28:0/1046722944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8fb40b81a0 0x7f8fb40cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 >> 172.21.15.28:0/1046722944 conn(0x7f8fb406ee30 msgr2=0x7f8fb415cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 shutdown_connections 2024-04-25T19:12:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.489+0000 7f8fbc54d640 1 -- 172.21.15.28:0/1046722944 wait complete. 2024-04-25T19:12:40.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.612+0000 7f6c1c37a640 1 Processor -- start 2024-04-25T19:12:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.612+0000 7f6c1c37a640 1 -- start start 2024-04-25T19:12:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.613+0000 7f6c1c37a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c140b5130 0x7f6c14161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.613+0000 7f6c1c37a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c1406ded0 0x7f6c1406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.613+0000 7f6c1c37a640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6c140d0830 0x7f6c140d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.613+0000 7f6c1c37a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f6c14163c30 con 0x7f6c140b5130 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.613+0000 7f6c1c37a640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f6c14164a90 con 0x7f6c140d0830 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.613+0000 7f6c1c37a640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1406e820 con 0x7f6c1406ded0 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a116640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c140b5130 0x7f6c14161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a116640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6c140b5130 0x7f6c14161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49586/0 (socket says 172.21.15.28:49586) 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a116640 1 -- 172.21.15.28:0/1576991550 learned_addr learned my addr 172.21.15.28:0/1576991550 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a917640 1 --2- 172.21.15.28:0/1576991550 >> v2:172.21.15.53:3300/0 conn(0x7f6c140d0830 0x7f6c140d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c19915640 1 --2- 172.21.15.28:0/1576991550 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c1406ded0 0x7f6c1406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a116640 1 -- 172.21.15.28:0/1576991550 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c1406ded0 msgr2=0x7f6c1406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a116640 1 --2- 172.21.15.28:0/1576991550 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c1406ded0 0x7f6c1406e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a116640 1 -- 172.21.15.28:0/1576991550 >> v2:172.21.15.53:3300/0 conn(0x7f6c140d0830 msgr2=0x7f6c140d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a116640 1 --2- 172.21.15.28:0/1576991550 >> v2:172.21.15.53:3300/0 conn(0x7f6c140d0830 0x7f6c140d0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a116640 1 -- 172.21.15.28:0/1576991550 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c14163e80 con 0x7f6c140b5130 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.614+0000 7f6c1a917640 1 --2- 172.21.15.28:0/1576991550 >> v2:172.21.15.53:3300/0 conn(0x7f6c140d0830 0x7f6c140d0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.615+0000 7f6c1a116640 1 --2- 172.21.15.28:0/1576991550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c140b5130 0x7f6c14161130 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6c1406edf0 tx=0x7f6c1002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd544a575c6c19c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.615+0000 7f6c19114640 1 -- 172.21.15.28:0/1576991550 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10037070 con 0x7f6c140b5130 2024-04-25T19:12:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.615+0000 7f6c19114640 1 -- 172.21.15.28:0/1576991550 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c10032440 con 0x7f6c140b5130 2024-04-25T19:12:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.615+0000 7f6c19114640 1 -- 172.21.15.28:0/1576991550 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10032810 con 0x7f6c140b5130 2024-04-25T19:12:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.615+0000 7f6c1c37a640 1 -- 172.21.15.28:0/1576991550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c140b5130 msgr2=0x7f6c14161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.615+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/1576991550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c140b5130 0x7f6c14161130 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6c1406edf0 tx=0x7f6c1002a1e0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.616+0000 7f6c1c37a640 1 -- 172.21.15.28:0/1576991550 shutdown_connections 2024-04-25T19:12:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.616+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/1576991550 >> v2:172.21.15.53:3300/0 conn(0x7f6c140d0830 0x7f6c140d0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.616+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/1576991550 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c1406ded0 0x7f6c1406e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.616+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/1576991550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c140b5130 0x7f6c14161130 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.616+0000 7f6c1c37a640 1 -- 172.21.15.28:0/1576991550 >> 172.21.15.28:0/1576991550 conn(0x7f6c140b7ad0 msgr2=0x7f6c140b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:40.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.616+0000 7f6c1c37a640 1 -- 172.21.15.28:0/1576991550 shutdown_connections 2024-04-25T19:12:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.616+0000 7f6c1c37a640 1 -- 172.21.15.28:0/1576991550 wait complete. 2024-04-25T19:12:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.617+0000 7f6c1c37a640 1 Processor -- start 2024-04-25T19:12:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.617+0000 7f6c1c37a640 1 -- start start 2024-04-25T19:12:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1c37a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1406ded0 0x7f6c140c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1c37a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c140b5130 0x7f6c1415bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1c37a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c140d0830 0x7f6c14162180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1c37a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c14164720 con 0x7f6c1406ded0 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1c37a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6c14164440 con 0x7f6c140b5130 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1c37a640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6c140d1180 con 0x7f6c140d0830 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1406ded0 0x7f6c140c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1406ded0 0x7f6c140c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49598/0 (socket says 172.21.15.28:49598) 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 -- 172.21.15.28:0/3554933945 learned_addr learned my addr 172.21.15.28:0/3554933945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c19915640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c140b5130 0x7f6c1415bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a917640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c140d0830 0x7f6c14162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 -- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c140d0830 msgr2=0x7f6c14162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c140d0830 0x7f6c14162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 -- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c140b5130 msgr2=0x7f6c1415bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c140b5130 0x7f6c1415bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 -- 172.21.15.28:0/3554933945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c140c8e10 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c19915640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c140b5130 0x7f6c1415bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a917640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c140d0830 0x7f6c14162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c1a116640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1406ded0 0x7f6c140c88d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6c1002a220 tx=0x7f6c100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.619+0000 7f6c0affd640 1 -- 172.21.15.28:0/3554933945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10041020 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.620+0000 7f6c0affd640 1 -- 172.21.15.28:0/3554933945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c100306c0 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.620+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1415a7a0 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.620+0000 7f6c0affd640 1 -- 172.21.15.28:0/3554933945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10037040 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.620+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1415ad60 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.620+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c14050740 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.621+0000 7f6c0affd640 1 -- 172.21.15.28:0/3554933945 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6c10044020 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.622+0000 7f6c0affd640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6bf80744b0 0x7f6bf80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.622+0000 7f6c0affd640 1 -- 172.21.15.28:0/3554933945 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f6c100b5880 con 0x7f6c1406ded0 2024-04-25T19:12:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.622+0000 7f6c19915640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6bf80744b0 0x7f6bf80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.622+0000 7f6c19915640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6bf80744b0 0x7f6bf80769c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6c14160b30 tx=0x7f6c04008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.626+0000 7f6c0affd640 1 -- 172.21.15.28:0/3554933945 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6c1004b030 con 0x7f6c1406ded0 2024-04-25T19:12:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.819+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c140d5290 con 0x7f6c1406ded0 2024-04-25T19:12:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.820+0000 7f6c0affd640 1 -- 172.21.15.28:0/3554933945 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+13104 (secure 0 0 0) 0x7f6c10080260 con 0x7f6c1406ded0 2024-04-25T19:12:40.822 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:40.822 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:37.652261+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6bf80744b0 msgr2=0x7f6bf80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6bf80744b0 0x7f6bf80769c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6c14160b30 tx=0x7f6c04008040 comp rx=0 tx=0).stop 2024-04-25T19:12:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1406ded0 msgr2=0x7f6c140c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1406ded0 0x7f6c140c88d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6c1002a220 tx=0x7f6c100046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 shutdown_connections 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6bf80744b0 0x7f6bf80769c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6c140d0830 0x7f6c14162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c140b5130 0x7f6c1415bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 --2- 172.21.15.28:0/3554933945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c1406ded0 0x7f6c140c88d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 >> 172.21.15.28:0/3554933945 conn(0x7f6c140b7ad0 msgr2=0x7f6c140d3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 shutdown_connections 2024-04-25T19:12:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.822+0000 7f6c1c37a640 1 -- 172.21.15.28:0/3554933945 wait complete. 2024-04-25T19:12:40.835 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:12:40.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.949+0000 7f4a9a50e640 1 Processor -- start 2024-04-25T19:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.949+0000 7f4a9a50e640 1 -- start start 2024-04-25T19:12:40.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.950+0000 7f4a9a50e640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4a940b9000 0x7f4a940d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.950+0000 7f4a9a50e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a9406e4a0 0x7f4a9406e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.950+0000 7f4a9a50e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a940b9830 con 0x7f4a940ce980 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.950+0000 7f4a9a50e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f4a940d2ea0 con 0x7f4a9406e4a0 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.950+0000 7f4a9a50e640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f4a940d0ec0 con 0x7f4a940b9000 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a937fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a9406e4a0 0x7f4a9406e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a98aab640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4a940ce980 0x7f4a940ced90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39650/0 (socket says 172.21.15.28:39650) 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a937fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4a9406e4a0 0x7f4a9406e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53938/0 (socket says 172.21.15.28:53938) 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a98aab640 1 -- 172.21.15.28:0/220201414 learned_addr learned my addr 172.21.15.28:0/220201414 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a93fff640 1 --2- 172.21.15.28:0/220201414 >> v2:172.21.15.53:3300/0 conn(0x7f4a940b9000 0x7f4a940d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a937fe640 1 -- 172.21.15.28:0/220201414 >> v2:172.21.15.53:3300/0 conn(0x7f4a940b9000 msgr2=0x7f4a940d3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a937fe640 1 --2- 172.21.15.28:0/220201414 >> v2:172.21.15.53:3300/0 conn(0x7f4a940b9000 0x7f4a940d3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a937fe640 1 -- 172.21.15.28:0/220201414 >> v1:172.21.15.28:6789/0 conn(0x7f4a940ce980 legacy=0x7f4a940ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a937fe640 1 -- 172.21.15.28:0/220201414 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9409d230 con 0x7f4a9406e4a0 2024-04-25T19:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.951+0000 7f4a92ffd640 1 -- 172.21.15.28:0/220201414 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146807115 0 0) 0x7f4a940b9830 con 0x7f4a940ce980 2024-04-25T19:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.952+0000 7f4a937fe640 1 --2- 172.21.15.28:0/220201414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9406e4a0 0x7f4a9406e8b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a9408cde0 tx=0x7f4a7c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=103de4815dd2a43f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.952+0000 7f4a92ffd640 1 -- 172.21.15.28:0/220201414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c037070 con 0x7f4a9406e4a0 2024-04-25T19:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.952+0000 7f4a92ffd640 1 -- 172.21.15.28:0/220201414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7c032440 con 0x7f4a9406e4a0 2024-04-25T19:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.952+0000 7f4a92ffd640 1 -- 172.21.15.28:0/220201414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c032810 con 0x7f4a9406e4a0 2024-04-25T19:12:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.953+0000 7f4a9a50e640 1 -- 172.21.15.28:0/220201414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9406e4a0 msgr2=0x7f4a9406e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.953+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/220201414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9406e4a0 0x7f4a9406e8b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4a9408cde0 tx=0x7f4a7c02a1a0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.953+0000 7f4a9a50e640 1 -- 172.21.15.28:0/220201414 shutdown_connections 2024-04-25T19:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.953+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/220201414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9406e4a0 0x7f4a9406e8b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.953+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/220201414 >> v2:172.21.15.53:3300/0 conn(0x7f4a940b9000 0x7f4a940d3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.953+0000 7f4a9a50e640 1 -- 172.21.15.28:0/220201414 >> 172.21.15.28:0/220201414 conn(0x7f4a940d03a0 msgr2=0x7f4a940d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.953+0000 7f4a9a50e640 1 -- 172.21.15.28:0/220201414 shutdown_connections 2024-04-25T19:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.953+0000 7f4a9a50e640 1 -- 172.21.15.28:0/220201414 wait complete. 2024-04-25T19:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.954+0000 7f4a9a50e640 1 Processor -- start 2024-04-25T19:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.955+0000 7f4a9a50e640 1 -- start start 2024-04-25T19:12:40.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a9a50e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9406e4a0 0x7f4a9415fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a9a50e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a940b9000 0x7f4a9415c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a9a50e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4a940ce980 0x7f4a9415d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a9a50e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9409cf50 con 0x7f4a940b9000 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a9a50e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9409cd00 con 0x7f4a9406e4a0 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a9a50e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4a9406d680 con 0x7f4a940ce980 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a940b9000 0x7f4a9415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a940b9000 0x7f4a9415c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53942/0 (socket says 172.21.15.28:53942) 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 -- 172.21.15.28:0/1326428882 learned_addr learned my addr 172.21.15.28:0/1326428882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 -- 172.21.15.28:0/1326428882 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4a940ce980 msgr2=0x7f4a9415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4a940ce980 0x7f4a9415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 -- 172.21.15.28:0/1326428882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9406e4a0 msgr2=0x7f4a9415fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a93fff640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9406e4a0 0x7f4a9415fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9406e4a0 0x7f4a9415fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 -- 172.21.15.28:0/1326428882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a940b18f0 con 0x7f4a940b9000 2024-04-25T19:12:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.956+0000 7f4a937fe640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a940b9000 0x7f4a9415c270 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c035040 tx=0x7f4a7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.957+0000 7f4a90ff9640 1 -- 172.21.15.28:0/1326428882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c037050 con 0x7f4a940b9000 2024-04-25T19:12:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.957+0000 7f4a90ff9640 1 -- 172.21.15.28:0/1326428882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a7c038040 con 0x7f4a940b9000 2024-04-25T19:12:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.957+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a940b1bd0 con 0x7f4a940b9000 2024-04-25T19:12:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.957+0000 7f4a90ff9640 1 -- 172.21.15.28:0/1326428882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c006e10 con 0x7f4a940b9000 2024-04-25T19:12:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.957+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a940b2190 con 0x7f4a940b9000 2024-04-25T19:12:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.958+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a94050740 con 0x7f4a940b9000 2024-04-25T19:12:40.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.959+0000 7f4a90ff9640 1 -- 172.21.15.28:0/1326428882 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4a7c030980 con 0x7f4a940b9000 2024-04-25T19:12:40.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.959+0000 7f4a90ff9640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4a740744e0 0x7f4a740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:40.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.959+0000 7f4a90ff9640 1 -- 172.21.15.28:0/1326428882 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f4a7c0b5bd0 con 0x7f4a940b9000 2024-04-25T19:12:40.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.959+0000 7f4a93fff640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4a740744e0 0x7f4a740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:40.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.959+0000 7f4a93fff640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4a740744e0 0x7f4a740769f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4a940d1760 tx=0x7f4a8400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:40.964+0000 7f4a90ff9640 1 -- 172.21.15.28:0/1326428882 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a7c080510 con 0x7f4a940b9000 2024-04-25T19:12:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.147+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a940b8be0 con 0x7f4a940b9000 2024-04-25T19:12:41.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.147+0000 7f4a90ff9640 1 -- 172.21.15.28:0/1326428882 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f4a7c07fe10 con 0x7f4a940b9000 2024-04-25T19:12:41.149 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4a740744e0 msgr2=0x7f4a740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4a740744e0 0x7f4a740769f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4a940d1760 tx=0x7f4a8400b040 comp rx=0 tx=0).stop 2024-04-25T19:12:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a940b9000 msgr2=0x7f4a9415c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a940b9000 0x7f4a9415c270 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c035040 tx=0x7f4a7c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 shutdown_connections 2024-04-25T19:12:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4a740744e0 0x7f4a740769f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4a940ce980 0x7f4a9415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a940b9000 0x7f4a9415c270 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 --2- 172.21.15.28:0/1326428882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9406e4a0 0x7f4a9415fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 >> 172.21.15.28:0/1326428882 conn(0x7f4a940d03a0 msgr2=0x7f4a9415cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 shutdown_connections 2024-04-25T19:12:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.149+0000 7f4a9a50e640 1 -- 172.21.15.28:0/1326428882 wait complete. 2024-04-25T19:12:41.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T19:12:41.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.272+0000 7f38196e2640 1 Processor -- start 2024-04-25T19:12:41.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.273+0000 7f38196e2640 1 -- start start 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f38196e2640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f381409d010 0x7f381406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f38196e2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381406d710 con 0x7f381409cb60 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f38196e2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f381408ce70 con 0x7f381406e410 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f38196e2640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f381408cc20 con 0x7f381409d010 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f38137fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f381409cb60 0x7f38140d1cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39656/0 (socket says 172.21.15.28:39656) 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f38137fe640 1 -- 172.21.15.28:0/3476779418 learned_addr learned my addr 172.21.15.28:0/3476779418 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f3812ffd640 1 --2- 172.21.15.28:0/3476779418 >> v2:172.21.15.53:3300/0 conn(0x7f381409d010 0x7f381406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1904889162 0 0) 0x7f381406d710 con 0x7f381409cb60 2024-04-25T19:12:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.274+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3800003520 con 0x7f381409cb60 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2901825196 0 0) 0x7f3800003520 con 0x7f381409cb60 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 >> v2:172.21.15.53:3300/0 conn(0x7f381409d010 msgr2=0x7f381406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 --2- 172.21.15.28:0/3476779418 >> v2:172.21.15.53:3300/0 conn(0x7f381409d010 0x7f381406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 >> v1:172.21.15.53:6789/0 conn(0x7f381406e410 legacy=0x7f38140cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38140b92d0 con 0x7f381409cb60 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f3804001670 con 0x7f381409cb60 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 >> v1:172.21.15.28:6789/0 conn(0x7f381409cb60 legacy=0x7f38140d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38000039b0 0x7f3800003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3800004800 0x7f3800004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3800005270 0x7f380000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f381408cc20 con 0x7f3800005270 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f380000c190 con 0x7f38000039b0 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f380000c390 con 0x7f3800004800 2024-04-25T19:12:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f38137fe640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3800005270 0x7f380000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f3812ffd640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38000039b0 0x7f3800003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f38127fc640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3800004800 0x7f3800004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f38137fe640 1 -- 172.21.15.28:0/3476779418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3800004800 msgr2=0x7f3800004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f38137fe640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3800004800 0x7f3800004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f38137fe640 1 -- 172.21.15.28:0/3476779418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38000039b0 msgr2=0x7f3800003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f38137fe640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38000039b0 0x7f3800003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.275+0000 7f38137fe640 1 -- 172.21.15.28:0/3476779418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3800003520 con 0x7f3800005270 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.276+0000 7f38137fe640 1 -- 172.21.15.28:0/3476779418 reap_dead start 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.276+0000 7f38137fe640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3800005270 0x7f380000b9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f380400c240 tx=0x7f3800005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e5ffcb2464e4c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.276+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3804007d10 con 0x7f3800005270 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.276+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3804005ad0 con 0x7f3800005270 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.276+0000 7f3811ffb640 1 -- 172.21.15.28:0/3476779418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3804007070 con 0x7f3800005270 2024-04-25T19:12:41.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.277+0000 7f38196e2640 1 -- 172.21.15.28:0/3476779418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3800005270 msgr2=0x7f380000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.277+0000 7f38196e2640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3800005270 0x7f380000b9c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f380400c240 tx=0x7f3800005830 comp rx=0 tx=0).stop 2024-04-25T19:12:41.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.277+0000 7f38196e2640 1 -- 172.21.15.28:0/3476779418 shutdown_connections 2024-04-25T19:12:41.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.277+0000 7f38196e2640 1 --2- 172.21.15.28:0/3476779418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3800005270 0x7f380000b9c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.277+0000 7f38196e2640 1 -- 172.21.15.28:0/3476779418 >> 172.21.15.28:0/3476779418 conn(0x7f38140d0430 msgr2=0x7f38140d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:41.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.277+0000 7f38196e2640 1 -- 172.21.15.28:0/3476779418 shutdown_connections 2024-04-25T19:12:41.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.277+0000 7f38196e2640 1 -- 172.21.15.28:0/3476779418 wait complete. 2024-04-25T19:12:41.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.279+0000 7f38196e2640 1 Processor -- start 2024-04-25T19:12:41.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.279+0000 7f38196e2640 1 -- start start 2024-04-25T19:12:41.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38196e2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800005270 0x7f38141f9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38196e2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38141f9c70 0x7f38141e9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38196e2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38141ea200 0x7f38141e3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38196e2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f381406eba0 con 0x7f38141f9c70 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38196e2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f381406ea20 con 0x7f3800005270 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38196e2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f381406ded0 con 0x7f38141ea200 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38127fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38141f9c70 0x7f38141e9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38127fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38141f9c70 0x7f38141e9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53956/0 (socket says 172.21.15.28:53956) 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.280+0000 7f38127fc640 1 -- 172.21.15.28:0/106757539 learned_addr learned my addr 172.21.15.28:0/106757539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f3812ffd640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800005270 0x7f38141f9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38137fe640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38141ea200 0x7f38141e3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38127fc640 1 -- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38141ea200 msgr2=0x7f38141e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38127fc640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38141ea200 0x7f38141e3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38127fc640 1 -- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800005270 msgr2=0x7f38141f9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38127fc640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800005270 0x7f38141f9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38127fc640 1 -- 172.21.15.28:0/106757539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38141ea610 con 0x7f38141f9c70 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f3812ffd640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800005270 0x7f38141f9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38137fe640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38141ea200 0x7f38141e3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38127fc640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38141f9c70 0x7f38141e9cc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00f5c0 tx=0x7f37fc00fa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f380b7fe640 1 -- 172.21.15.28:0/106757539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc0052e0 con 0x7f38141f9c70 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f380b7fe640 1 -- 172.21.15.28:0/106757539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37fc018070 con 0x7f38141f9c70 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38141f61c0 con 0x7f38141f9c70 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f380b7fe640 1 -- 172.21.15.28:0/106757539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc003830 con 0x7f38141f9c70 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.281+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38141f6780 con 0x7f38141f9c70 2024-04-25T19:12:41.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.282+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38141f6f80 con 0x7f38141f9c70 2024-04-25T19:12:41.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.283+0000 7f380b7fe640 1 -- 172.21.15.28:0/106757539 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f37fc0039d0 con 0x7f38141f9c70 2024-04-25T19:12:41.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.283+0000 7f380b7fe640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f37f40744b0 0x7f37f40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.284+0000 7f380b7fe640 1 -- 172.21.15.28:0/106757539 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f37fc094e00 con 0x7f38141f9c70 2024-04-25T19:12:41.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.284+0000 7f3812ffd640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f37f40744b0 0x7f37f40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.284+0000 7f3812ffd640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f37f40744b0 0x7f37f40769c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f380c007520 tx=0x7f380c00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:41.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.288+0000 7f380b7fe640 1 -- 172.21.15.28:0/106757539 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38141ea610 con 0x7f38141f9c70 2024-04-25T19:12:41.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.477+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3814050740 con 0x7f38141f9c70 2024-04-25T19:12:41.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.477+0000 7f380b7fe640 1 -- 172.21.15.28:0/106757539 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) ==== 84+0+65 (secure 0 0 0) 0x7f37fc05f7d0 con 0x7f38141f9c70 2024-04-25T19:12:41.479 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:41.479 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-04-25T19:12:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-25T19:12:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-04-25T19:12:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-04-25T19:12:41.480 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:12:41.480 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-04-25T19:12:41.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f37f40744b0 msgr2=0x7f37f40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f37f40744b0 0x7f37f40769c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f380c007520 tx=0x7f380c00b040 comp rx=0 tx=0).stop 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38141f9c70 msgr2=0x7f38141e9cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38141f9c70 0x7f38141e9cc0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00f5c0 tx=0x7f37fc00fa80 comp rx=0 tx=0).stop 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 shutdown_connections 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f37f40744b0 0x7f37f40769c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38141ea200 0x7f38141e3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38141f9c70 0x7f38141e9cc0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 --2- 172.21.15.28:0/106757539 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3800005270 0x7f38141f9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 >> 172.21.15.28:0/106757539 conn(0x7f38140d0430 msgr2=0x7f38141eac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 shutdown_connections 2024-04-25T19:12:41.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.479+0000 7f38196e2640 1 -- 172.21.15.28:0/106757539 wait complete. 2024-04-25T19:12:41.492 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T19:12:41.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T19:12:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.607+0000 7f4ea8f5f640 1 Processor -- start 2024-04-25T19:12:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.607+0000 7f4ea8f5f640 1 -- start start 2024-04-25T19:12:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.608+0000 7f4ea8f5f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ea408cbb0 0x7f4ea4162620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.608+0000 7f4ea8f5f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ea40cf800 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.608+0000 7f4ea8f5f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ea409d170 con 0x7f4ea4162b60 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.608+0000 7f4ea8f5f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f4ea40b43b0 con 0x7f4ea408cbb0 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.608+0000 7f4e9bfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ea408cbb0 0x7f4ea4162620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.608+0000 7f4ea259c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4ea40b8eb0 0x7f4ea409bd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39670/0 (socket says 172.21.15.28:39670) 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.608+0000 7f4e9bfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4ea408cbb0 0x7f4ea4162620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53958/0 (socket says 172.21.15.28:53958) 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.608+0000 7f4ea259c640 1 -- 172.21.15.28:0/3904288778 learned_addr learned my addr 172.21.15.28:0/3904288778 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.609+0000 7f4e9bfff640 1 -- 172.21.15.28:0/3904288778 >> v1:172.21.15.53:6789/0 conn(0x7f4ea4162b60 legacy=0x7f4ea40b67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.609+0000 7f4e9bfff640 1 -- 172.21.15.28:0/3904288778 >> v1:172.21.15.28:6789/0 conn(0x7f4ea40b8eb0 legacy=0x7f4ea409bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.609+0000 7f4e9bfff640 1 -- 172.21.15.28:0/3904288778 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea40b4e00 con 0x7f4ea408cbb0 2024-04-25T19:12:41.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.609+0000 7f4ea1d9b640 1 -- 172.21.15.28:0/3904288778 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2046067916 0 0) 0x7f4ea40cf800 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.609+0000 7f4e9bfff640 1 --2- 172.21.15.28:0/3904288778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea408cbb0 0x7f4ea4162620 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0097a0 tx=0x7f4e8c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c45d44007cccb36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea1d9b640 1 -- 172.21.15.28:0/3904288778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c037070 con 0x7f4ea408cbb0 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea1d9b640 1 -- 172.21.15.28:0/3904288778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8c032440 con 0x7f4ea408cbb0 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea1d9b640 1 -- 172.21.15.28:0/3904288778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c032810 con 0x7f4ea408cbb0 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/3904288778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea408cbb0 msgr2=0x7f4ea4162620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea8f5f640 1 --2- 172.21.15.28:0/3904288778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea408cbb0 0x7f4ea4162620 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0097a0 tx=0x7f4e8c02a1e0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/3904288778 shutdown_connections 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea8f5f640 1 --2- 172.21.15.28:0/3904288778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea408cbb0 0x7f4ea4162620 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/3904288778 >> 172.21.15.28:0/3904288778 conn(0x7f4ea40b5470 msgr2=0x7f4ea40b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/3904288778 shutdown_connections 2024-04-25T19:12:41.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.610+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/3904288778 wait complete. 2024-04-25T19:12:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.612+0000 7f4ea8f5f640 1 Processor -- start 2024-04-25T19:12:41.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.612+0000 7f4ea8f5f640 1 -- start start 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea8f5f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ea408cbb0 0x7f4ea4158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea8f5f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea40b8eb0 0x7f4ea4156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea8f5f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ea4162b60 0x7f4ea406e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea8f5f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea40cf2e0 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea8f5f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4ea409cf20 con 0x7f4ea4162b60 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea8f5f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4ea409c440 con 0x7f4ea408cbb0 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea40b8eb0 0x7f4ea4156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea40b8eb0 0x7f4ea4156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53974/0 (socket says 172.21.15.28:53974) 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 -- 172.21.15.28:0/1570849957 learned_addr learned my addr 172.21.15.28:0/1570849957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea2d9d640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ea4162b60 0x7f4ea406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea259c640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ea408cbb0 0x7f4ea4158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 -- 172.21.15.28:0/1570849957 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ea408cbb0 msgr2=0x7f4ea4158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ea408cbb0 0x7f4ea4158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 -- 172.21.15.28:0/1570849957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ea4162b60 msgr2=0x7f4ea406e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ea4162b60 0x7f4ea406e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 -- 172.21.15.28:0/1570849957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ea415ae20 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9bfff640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea40b8eb0 0x7f4ea4156b90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0098c0 tx=0x7f4e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9affd640 1 -- 172.21.15.28:0/1570849957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c041070 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4e9affd640 1 -- 172.21.15.28:0/1570849957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8c03c930 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.614+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ea40b22a0 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.615+0000 7f4e9affd640 1 -- 172.21.15.28:0/1570849957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c037040 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.615+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ea41570d0 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.615+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ea406a100 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.616+0000 7f4e9affd640 1 -- 172.21.15.28:0/1570849957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4e8c044070 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.617+0000 7f4e9affd640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4e840744e0 0x7f4e840769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.617+0000 7f4ea259c640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4e840744e0 0x7f4e840769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.617+0000 7f4e9affd640 1 -- 172.21.15.28:0/1570849957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f4e8c0b5ba0 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.617+0000 7f4ea259c640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4e840744e0 0x7f4e840769f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4e94007fd0 tx=0x7f4e94007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:41.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.621+0000 7f4e9affd640 1 -- 172.21.15.28:0/1570849957 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e8c07fea0 con 0x7f4ea40b8eb0 2024-04-25T19:12:41.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:41.805+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f4ea409c7f0 con 0x7f4ea40b8eb0 2024-04-25T19:12:42.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.660+0000 7f4e9affd640 1 -- 172.21.15.28:0/1570849957 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) ==== 138+0+0 (secure 0 0 0) 0x7f4e8c0841a0 con 0x7f4ea40b8eb0 2024-04-25T19:12:42.662 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-04-25T19:12:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.663+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4e840744e0 msgr2=0x7f4e840769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.663+0000 7f4ea8f5f640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4e840744e0 0x7f4e840769f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4e94007fd0 tx=0x7f4e94007de0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.663+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea40b8eb0 msgr2=0x7f4ea4156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.663+0000 7f4ea8f5f640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea40b8eb0 0x7f4ea4156b90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0098c0 tx=0x7f4e8c0046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.664+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 shutdown_connections 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.664+0000 7f4ea8f5f640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4e840744e0 0x7f4e840769f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.664+0000 7f4ea8f5f640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ea4162b60 0x7f4ea406e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.664+0000 7f4ea8f5f640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ea40b8eb0 0x7f4ea4156b90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.664+0000 7f4ea8f5f640 1 --2- 172.21.15.28:0/1570849957 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4ea408cbb0 0x7f4ea4158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.664+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 >> 172.21.15.28:0/1570849957 conn(0x7f4ea40b5470 msgr2=0x7f4ea40b8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.664+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 shutdown_connections 2024-04-25T19:12:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.664+0000 7f4ea8f5f640 1 -- 172.21.15.28:0/1570849957 wait complete. 2024-04-25T19:12:42.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T19:12:42.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.807+0000 7f9472a92640 1 Processor -- start 2024-04-25T19:12:42.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.807+0000 7f9472a92640 1 -- start start 2024-04-25T19:12:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f9472a92640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f946c0d25b0 0x7f946c0d29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f9472a92640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c0d4480 0x7f946c0d4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f9472a92640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f946c0d2f00 con 0x7f946c0b4e10 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f9472a92640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f946c161150 con 0x7f946c0d25b0 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f9472a92640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f946c161430 con 0x7f946c0d4480 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f947082e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f946c0b4e10 0x7f946c164650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39684/0 (socket says 172.21.15.28:39684) 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f947082e640 1 -- 172.21.15.28:0/3702625330 learned_addr learned my addr 172.21.15.28:0/3702625330 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2198579250 0 0) 0x7f9460001110 con 0x7f946c0b4e10 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f947102f640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c0d4480 0x7f946c0d4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f945c003520 con 0x7f946c0b4e10 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946bfff640 1 --2- 172.21.15.28:0/3702625330 >> v2:172.21.15.53:3300/0 conn(0x7f946c0d25b0 0x7f946c0d29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2058296080 0 0) 0x7f945c003520 con 0x7f946c0b4e10 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c0d4480 msgr2=0x7f946c0d4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c0d4480 0x7f946c0d4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 >> v2:172.21.15.53:3300/0 conn(0x7f946c0d25b0 msgr2=0x7f946c0d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 --2- 172.21.15.28:0/3702625330 >> v2:172.21.15.53:3300/0 conn(0x7f946c0d25b0 0x7f946c0d29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c163e50 con 0x7f946c0b4e10 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f9460001840 con 0x7f946c0b4e10 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 >> v1:172.21.15.28:6789/0 conn(0x7f946c0b4e10 legacy=0x7f946c164650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c0d4480 0x7f945c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f945c003f30 0x7f945c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f945c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f946c161150 con 0x7f945c004d40 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f946c161430 con 0x7f945c003f30 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.809+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f945c00bac0 con 0x7f946c0d4480 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.810+0000 7f947102f640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f945c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.810+0000 7f947082e640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c0d4480 0x7f945c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.810+0000 7f946bfff640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f945c003f30 0x7f945c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.810+0000 7f947102f640 1 -- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c0d4480 msgr2=0x7f945c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.810+0000 7f947102f640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c0d4480 0x7f945c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.810+0000 7f947102f640 1 -- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f945c003f30 msgr2=0x7f945c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.810+0000 7f947102f640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f945c003f30 0x7f945c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.810+0000 7f947102f640 1 -- 172.21.15.28:0/3702625330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f945c003520 con 0x7f945c004d40 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.811+0000 7f947102f640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f945c005150 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9464009fc0 tx=0x7f945c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=352ac864c2ea5608 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.811+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9464034020 con 0x7f945c004d40 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.811+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94640305f0 con 0x7f945c004d40 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.811+0000 7f946b7fe640 1 -- 172.21.15.28:0/3702625330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9464009c10 con 0x7f945c004d40 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.811+0000 7f9472a92640 1 -- 172.21.15.28:0/3702625330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 msgr2=0x7f945c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.811+0000 7f9472a92640 1 --2- 172.21.15.28:0/3702625330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f945c005150 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9464009fc0 tx=0x7f945c005830 comp rx=0 tx=0).stop 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.812+0000 7f947102f640 1 -- 172.21.15.28:0/3702625330 reap_dead start 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.812+0000 7f9472a92640 1 -- 172.21.15.28:0/3702625330 shutdown_connections 2024-04-25T19:12:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.812+0000 7f9472a92640 1 -- 172.21.15.28:0/3702625330 >> 172.21.15.28:0/3702625330 conn(0x7f946c06ee10 msgr2=0x7f946c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.812+0000 7f9472a92640 1 -- 172.21.15.28:0/3702625330 shutdown_connections 2024-04-25T19:12:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.812+0000 7f9472a92640 1 -- 172.21.15.28:0/3702625330 wait complete. 2024-04-25T19:12:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.813+0000 7f9472a92640 1 Processor -- start 2024-04-25T19:12:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.813+0000 7f9472a92640 1 -- start start 2024-04-25T19:12:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f9472a92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f946c1590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f9472a92640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c162090 0x7f946c1ea390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f9472a92640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f946c1ea8d0 0x7f946c0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f9472a92640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f946c0d4890 con 0x7f945c004d40 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f9472a92640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f946c0b7850 con 0x7f946c1ea8d0 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f9472a92640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f946c0d4a10 con 0x7f946c162090 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947082e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f946c1590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947082e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f946c1590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53990/0 (socket says 172.21.15.28:53990) 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947082e640 1 -- 172.21.15.28:0/2523991791 learned_addr learned my addr 172.21.15.28:0/2523991791 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947102f640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f946c1ea8d0 0x7f946c0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f946bfff640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c162090 0x7f946c1ea390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947082e640 1 -- 172.21.15.28:0/2523991791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c162090 msgr2=0x7f946c1ea390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947082e640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c162090 0x7f946c1ea390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947082e640 1 -- 172.21.15.28:0/2523991791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f946c1ea8d0 msgr2=0x7f946c0ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947082e640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f946c1ea8d0 0x7f946c0ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.815+0000 7f947082e640 1 -- 172.21.15.28:0/2523991791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f946c1eace0 con 0x7f945c004d40 2024-04-25T19:12:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.816+0000 7f947082e640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f946c1590e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9460007310 tx=0x7f94600077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.816+0000 7f94697fa640 1 -- 172.21.15.28:0/2523991791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946000f9c0 con 0x7f945c004d40 2024-04-25T19:12:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.816+0000 7f94697fa640 1 -- 172.21.15.28:0/2523991791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9460018070 con 0x7f945c004d40 2024-04-25T19:12:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.816+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f946c0b7600 con 0x7f945c004d40 2024-04-25T19:12:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.816+0000 7f94697fa640 1 -- 172.21.15.28:0/2523991791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946000fd80 con 0x7f945c004d40 2024-04-25T19:12:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.816+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f946c0b7040 con 0x7f945c004d40 2024-04-25T19:12:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.817+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f946c050740 con 0x7f945c004d40 2024-04-25T19:12:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.818+0000 7f94697fa640 1 -- 172.21.15.28:0/2523991791 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f946001d460 con 0x7f945c004d40 2024-04-25T19:12:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.818+0000 7f94697fa640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f94500744b0 0x7f94500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.819+0000 7f946bfff640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f94500744b0 0x7f94500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.819+0000 7f94697fa640 1 -- 172.21.15.28:0/2523991791 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5619+0+0 (secure 0 0 0) 0x7f9460095e50 con 0x7f945c004d40 2024-04-25T19:12:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.819+0000 7f946bfff640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f94500744b0 0x7f94500769c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f946c0bae10 tx=0x7f9458008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:42.823+0000 7f94697fa640 1 -- 172.21.15.28:0/2523991791 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f946c1eace0 con 0x7f945c004d40 2024-04-25T19:12:43.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.009+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f946c163740 con 0x7f945c004d40 2024-04-25T19:12:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.666+0000 7f94697fa640 1 -- 172.21.15.28:0/2523991791 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) ==== 170+0+0 (secure 0 0 0) 0x7f946c0d2f00 con 0x7f945c004d40 2024-04-25T19:12:43.668 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-04-25T19:12:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f94500744b0 msgr2=0x7f94500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:43.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f94500744b0 0x7f94500769c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f946c0bae10 tx=0x7f9458008040 comp rx=0 tx=0).stop 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 msgr2=0x7f946c1590e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f946c1590e0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9460007310 tx=0x7f94600077d0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 shutdown_connections 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f94500744b0 0x7f94500769c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f946c1ea8d0 0x7f946c0ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f946c162090 0x7f946c1ea390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 --2- 172.21.15.28:0/2523991791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f945c004d40 0x7f946c1590e0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 >> 172.21.15.28:0/2523991791 conn(0x7f946c06ee10 msgr2=0x7f946c15fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 shutdown_connections 2024-04-25T19:12:43.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.669+0000 7f9472a92640 1 -- 172.21.15.28:0/2523991791 wait complete. 2024-04-25T19:12:43.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T19:12:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.821+0000 7f16e4f2d640 1 Processor -- start 2024-04-25T19:12:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.821+0000 7f16e4f2d640 1 -- start start 2024-04-25T19:12:43.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16e4f2d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16e01696c0 0x7f16e016bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16e4f2d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e00b92a0 con 0x7f16e00b3d00 2024-04-25T19:12:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16e4f2d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16e009cea0 con 0x7f16e00d2d10 2024-04-25T19:12:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16e4f2d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f16e00d28d0 con 0x7f16e01696c0 2024-04-25T19:12:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16ddd9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f16e00b3d00 0x7f16e00b4110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39686/0 (socket says 172.21.15.28:39686) 2024-04-25T19:12:43.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16ddd9b640 1 -- 172.21.15.28:0/876267067 learned_addr learned my addr 172.21.15.28:0/876267067 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:43.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 571595626 0 0) 0x7f16e00b92a0 con 0x7f16e00b3d00 2024-04-25T19:12:43.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16c8003520 con 0x7f16e00b3d00 2024-04-25T19:12:43.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16ded9d640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16e01696c0 0x7f16e016bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:43.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3215162049 0 0) 0x7f16c8003520 con 0x7f16e00b3d00 2024-04-25T19:12:43.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16e01696c0 msgr2=0x7f16e016bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:43.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16e01696c0 0x7f16e016bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 >> v1:172.21.15.53:6789/0 conn(0x7f16e00d2d10 legacy=0x7f16e00d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:43.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e006e8e0 con 0x7f16e00b3d00 2024-04-25T19:12:43.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f16d4001670 con 0x7f16e00b3d00 2024-04-25T19:12:43.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 >> v1:172.21.15.28:6789/0 conn(0x7f16e00b3d00 legacy=0x7f16e00b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c80039e0 0x7f16c8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16c8004800 0x7f16c8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16e01696c0 0x7f16c8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16e00d28d0 con 0x7f16c80039e0 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f16c800bb20 con 0x7f16c8004800 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.823+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f16c800bd20 con 0x7f16e01696c0 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.824+0000 7f16de59c640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c80039e0 0x7f16c8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.824+0000 7f16ddd9b640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16c8004800 0x7f16c8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.824+0000 7f16ded9d640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16e01696c0 0x7f16c8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.824+0000 7f16de59c640 1 -- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16e01696c0 msgr2=0x7f16c8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.824+0000 7f16de59c640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16e01696c0 0x7f16c8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.824+0000 7f16de59c640 1 -- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16c8004800 msgr2=0x7f16c8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.824+0000 7f16de59c640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16c8004800 0x7f16c8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.824+0000 7f16de59c640 1 -- 172.21.15.28:0/876267067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c8003520 con 0x7f16c80039e0 2024-04-25T19:12:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16de59c640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c80039e0 0x7f16c8003e10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f16d0001830 tx=0x7f16c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaeae7671c535548 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0037070 con 0x7f16c80039e0 2024-04-25T19:12:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d002abf0 con 0x7f16c80039e0 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16dd59a640 1 -- 172.21.15.28:0/876267067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0001430 con 0x7f16c80039e0 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16e4f2d640 1 -- 172.21.15.28:0/876267067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c80039e0 msgr2=0x7f16c8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16e4f2d640 1 --2- 172.21.15.28:0/876267067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c80039e0 0x7f16c8003e10 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f16d0001830 tx=0x7f16c8005830 comp rx=0 tx=0).stop 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16ded9d640 1 -- 172.21.15.28:0/876267067 reap_dead start 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16e4f2d640 1 -- 172.21.15.28:0/876267067 shutdown_connections 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.825+0000 7f16e4f2d640 1 -- 172.21.15.28:0/876267067 >> 172.21.15.28:0/876267067 conn(0x7f16e00d0450 msgr2=0x7f16e00d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.826+0000 7f16e4f2d640 1 -- 172.21.15.28:0/876267067 shutdown_connections 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.826+0000 7f16e4f2d640 1 -- 172.21.15.28:0/876267067 wait complete. 2024-04-25T19:12:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.827+0000 7f16e4f2d640 1 Processor -- start 2024-04-25T19:12:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.827+0000 7f16e4f2d640 1 -- start start 2024-04-25T19:12:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16e4f2d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16c80039e0 0x7f16e01fa060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16e4f2d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e015b830 0x7f16e0161bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16e4f2d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16e01620f0 0x7f16e015fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16e4f2d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16e006ebc0 con 0x7f16e01620f0 2024-04-25T19:12:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16e4f2d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f16e006e690 con 0x7f16e015b830 2024-04-25T19:12:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16e4f2d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f16e00d3120 con 0x7f16c80039e0 2024-04-25T19:12:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16e01620f0 0x7f16e015fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16e01620f0 0x7f16e015fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54010/0 (socket says 172.21.15.28:54010) 2024-04-25T19:12:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 -- 172.21.15.28:0/2052564517 learned_addr learned my addr 172.21.15.28:0/2052564517 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16de59c640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16c80039e0 0x7f16e01fa060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ddd9b640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e015b830 0x7f16e0161bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 -- 172.21.15.28:0/2052564517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16c80039e0 msgr2=0x7f16e01fa060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16c80039e0 0x7f16e01fa060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 -- 172.21.15.28:0/2052564517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e015b830 msgr2=0x7f16e0161bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e015b830 0x7f16e0161bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 -- 172.21.15.28:0/2052564517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e0162500 con 0x7f16e01620f0 2024-04-25T19:12:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.829+0000 7f16ded9d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16e01620f0 0x7f16e015fbc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f16e008d040 tx=0x7f16d800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.830+0000 7f16ceffd640 1 -- 172.21.15.28:0/2052564517 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d8015430 con 0x7f16e01620f0 2024-04-25T19:12:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.830+0000 7f16ceffd640 1 -- 172.21.15.28:0/2052564517 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d801d070 con 0x7f16e01620f0 2024-04-25T19:12:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.830+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e00c74e0 con 0x7f16e01620f0 2024-04-25T19:12:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.830+0000 7f16ceffd640 1 -- 172.21.15.28:0/2052564517 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d8015810 con 0x7f16e01620f0 2024-04-25T19:12:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.830+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e00ca110 con 0x7f16e01620f0 2024-04-25T19:12:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.831+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16e00b0910 con 0x7f16e01620f0 2024-04-25T19:12:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.832+0000 7f16ceffd640 1 -- 172.21.15.28:0/2052564517 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f16d8005d10 con 0x7f16e01620f0 2024-04-25T19:12:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.832+0000 7f16ceffd640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f16c00744b0 0x7f16c00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.832+0000 7f16de59c640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f16c00744b0 0x7f16c00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.832+0000 7f16ceffd640 1 -- 172.21.15.28:0/2052564517 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6001+0+0 (secure 0 0 0) 0x7f16d80966d0 con 0x7f16e01620f0 2024-04-25T19:12:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.833+0000 7f16de59c640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f16c00744b0 0x7f16c00769c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f16d0034000 tx=0x7f16d0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:43.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:43.837+0000 7f16ceffd640 1 -- 172.21.15.28:0/2052564517 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f16e0162500 con 0x7f16e01620f0 2024-04-25T19:12:44.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.061+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f16e0050740 con 0x7f16e01620f0 2024-04-25T19:12:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.667+0000 7f16ceffd640 1 -- 172.21.15.28:0/2052564517 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7f16d8010040 con 0x7f16e01620f0 2024-04-25T19:12:44.669 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 5 and data pool 6 2024-04-25T19:12:44.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f16c00744b0 msgr2=0x7f16c00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f16c00744b0 0x7f16c00769c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f16d0034000 tx=0x7f16d0009030 comp rx=0 tx=0).stop 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16e01620f0 msgr2=0x7f16e015fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16e01620f0 0x7f16e015fbc0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f16e008d040 tx=0x7f16d800f580 comp rx=0 tx=0).stop 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 shutdown_connections 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f16c00744b0 0x7f16c00769c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16e01620f0 0x7f16e015fbc0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16e015b830 0x7f16e0161bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 --2- 172.21.15.28:0/2052564517 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f16c80039e0 0x7f16e01fa060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 >> 172.21.15.28:0/2052564517 conn(0x7f16e00d0450 msgr2=0x7f16e00d21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.671+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 shutdown_connections 2024-04-25T19:12:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.672+0000 7f16e4f2d640 1 -- 172.21.15.28:0/2052564517 wait complete. 2024-04-25T19:12:44.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.819+0000 7f98de346640 1 Processor -- start 2024-04-25T19:12:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.819+0000 7f98de346640 1 -- start start 2024-04-25T19:12:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.820+0000 7f98de346640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f98d809cf00 0x7f98d80b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.820+0000 7f98de346640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d80ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.820+0000 7f98de346640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98d80d1ce0 con 0x7f98d809caf0 2024-04-25T19:12:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.820+0000 7f98de346640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f98d80b8fa0 con 0x7f98d809cf00 2024-04-25T19:12:44.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.820+0000 7f98de346640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f98d80cf8d0 con 0x7f98d80b7710 2024-04-25T19:12:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.820+0000 7f98dc8e3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d80ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.820+0000 7f98d77fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f98d809cf00 0x7f98d80b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.820+0000 7f98d7fff640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f98d809caf0 0x7f98d80d2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:41540/0 (socket says 172.21.15.28:41540) 2024-04-25T19:12:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.821+0000 7f98dc8e3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d80ce720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:55438/0 (socket says 172.21.15.28:55438) 2024-04-25T19:12:44.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.821+0000 7f98d7fff640 1 -- 172.21.15.28:0/3976182201 learned_addr learned my addr 172.21.15.28:0/3976182201 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.821+0000 7f98dc8e3640 1 -- 172.21.15.28:0/3976182201 >> v2:172.21.15.53:3300/0 conn(0x7f98d809cf00 msgr2=0x7f98d80b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.821+0000 7f98dc8e3640 1 --2- 172.21.15.28:0/3976182201 >> v2:172.21.15.53:3300/0 conn(0x7f98d809cf00 0x7f98d80b71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.821+0000 7f98dc8e3640 1 -- 172.21.15.28:0/3976182201 >> v1:172.21.15.53:6789/0 conn(0x7f98d809caf0 legacy=0x7f98d80d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.821+0000 7f98dc8e3640 1 -- 172.21.15.28:0/3976182201 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d80cfbb0 con 0x7f98d80b7710 2024-04-25T19:12:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.821+0000 7f98d77fe640 1 --2- 172.21.15.28:0/3976182201 >> v2:172.21.15.53:3300/0 conn(0x7f98d809cf00 0x7f98d80b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:12:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.822+0000 7f98dc8e3640 1 --2- 172.21.15.28:0/3976182201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d80ce720 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009d90 tx=0x7f98d002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=534ec22ee2238aa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.822+0000 7f98d6ffd640 1 -- 172.21.15.28:0/3976182201 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0031600 con 0x7f98d80b7710 2024-04-25T19:12:44.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.822+0000 7f98d6ffd640 1 -- 172.21.15.28:0/3976182201 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d00317a0 con 0x7f98d80b7710 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.822+0000 7f98d6ffd640 1 -- 172.21.15.28:0/3976182201 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0031b70 con 0x7f98d80b7710 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.823+0000 7f98de346640 1 -- 172.21.15.28:0/3976182201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 msgr2=0x7f98d80ce720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.823+0000 7f98de346640 1 --2- 172.21.15.28:0/3976182201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d80ce720 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f98d0009d90 tx=0x7f98d002a5d0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.823+0000 7f98de346640 1 -- 172.21.15.28:0/3976182201 shutdown_connections 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.823+0000 7f98de346640 1 --2- 172.21.15.28:0/3976182201 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d80ce720 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.823+0000 7f98de346640 1 --2- 172.21.15.28:0/3976182201 >> v2:172.21.15.53:3300/0 conn(0x7f98d809cf00 0x7f98d80b71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.823+0000 7f98de346640 1 -- 172.21.15.28:0/3976182201 >> 172.21.15.28:0/3976182201 conn(0x7f98d80d03f0 msgr2=0x7f98d80d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.823+0000 7f98de346640 1 -- 172.21.15.28:0/3976182201 shutdown_connections 2024-04-25T19:12:44.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.823+0000 7f98de346640 1 -- 172.21.15.28:0/3976182201 wait complete. 2024-04-25T19:12:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.824+0000 7f98de346640 1 Processor -- start 2024-04-25T19:12:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.825+0000 7f98de346640 1 -- start start 2024-04-25T19:12:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98de346640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d809caf0 0x7f98d81d8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98de346640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d809cf00 0x7f98d809ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98de346640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d81e6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98de346640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98d80b9430 con 0x7f98d809caf0 2024-04-25T19:12:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98de346640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f98d80d19e0 con 0x7f98d809cf00 2024-04-25T19:12:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98de346640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f98d80d4790 con 0x7f98d80b7710 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d809caf0 0x7f98d81d8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d809caf0 0x7f98d81d8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54016/0 (socket says 172.21.15.28:54016) 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d7fff640 1 -- 172.21.15.28:0/54755483 learned_addr learned my addr 172.21.15.28:0/54755483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98dc8e3640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d81e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d77fe640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d809cf00 0x7f98d809ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d7fff640 1 -- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 msgr2=0x7f98d81e6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d7fff640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d81e6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d7fff640 1 -- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d809cf00 msgr2=0x7f98d809ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d7fff640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d809cf00 0x7f98d809ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d7fff640 1 -- 172.21.15.28:0/54755483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d81d9060 con 0x7f98d809caf0 2024-04-25T19:12:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98dc8e3640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d81e6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.826+0000 7f98d77fe640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d809cf00 0x7f98d809ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.827+0000 7f98d7fff640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d809caf0 0x7f98d81d8b20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f98c800c3c0 tx=0x7f98c800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.827+0000 7f98d4ff9640 1 -- 172.21.15.28:0/54755483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c8017070 con 0x7f98d809caf0 2024-04-25T19:12:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.827+0000 7f98d4ff9640 1 -- 172.21.15.28:0/54755483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98c8005ce0 con 0x7f98d809caf0 2024-04-25T19:12:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.827+0000 7f98d4ff9640 1 -- 172.21.15.28:0/54755483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c80072f0 con 0x7f98d809caf0 2024-04-25T19:12:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.827+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98d81f45a0 con 0x7f98d809caf0 2024-04-25T19:12:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.827+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98d81e9b70 con 0x7f98d809caf0 2024-04-25T19:12:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.828+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98d8050740 con 0x7f98d809caf0 2024-04-25T19:12:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.829+0000 7f98d4ff9640 1 -- 172.21.15.28:0/54755483 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f98c801a070 con 0x7f98d809caf0 2024-04-25T19:12:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.829+0000 7f98d4ff9640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98c40744b0 0x7f98c40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.829+0000 7f98d4ff9640 1 -- 172.21.15.28:0/54755483 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f98c80947f0 con 0x7f98d809caf0 2024-04-25T19:12:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.829+0000 7f98d77fe640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98c40744b0 0x7f98c40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.830+0000 7f98d77fe640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98c40744b0 0x7f98c40769c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f98d80cc210 tx=0x7f98cc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:44.834+0000 7f98d4ff9640 1 -- 172.21.15.28:0/54755483 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f98d81d9060 con 0x7f98d809caf0 2024-04-25T19:12:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.021+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98d80b7e20 con 0x7f98d809caf0 2024-04-25T19:12:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.022+0000 7f98d4ff9640 1 -- 172.21.15.28:0/54755483 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16895 (secure 0 0 0) 0x7f98c8065e20 con 0x7f98d809caf0 2024-04-25T19:12:45.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:45.024 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:44.063334+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:45.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98c40744b0 msgr2=0x7f98c40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98c40744b0 0x7f98c40769c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f98d80cc210 tx=0x7f98cc008040 comp rx=0 tx=0).stop 2024-04-25T19:12:45.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d809caf0 msgr2=0x7f98d81d8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d809caf0 0x7f98d81d8b20 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f98c800c3c0 tx=0x7f98c800c7c0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 shutdown_connections 2024-04-25T19:12:45.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98c40744b0 0x7f98c40769c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d80b7710 0x7f98d81e6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d809cf00 0x7f98d809ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 --2- 172.21.15.28:0/54755483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d809caf0 0x7f98d81d8b20 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 >> 172.21.15.28:0/54755483 conn(0x7f98d80d03f0 msgr2=0x7f98d80d2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:45.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 shutdown_connections 2024-04-25T19:12:45.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.024+0000 7f98de346640 1 -- 172.21.15.28:0/54755483 wait complete. 2024-04-25T19:12:45.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.147+0000 7f198c227640 1 Processor -- start 2024-04-25T19:12:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.147+0000 7f198c227640 1 -- start start 2024-04-25T19:12:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.148+0000 7f198c227640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b5130 0x7f1984161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.148+0000 7f198c227640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f198415f7d0 0x7f198415fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.148+0000 7f198c227640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1984163e80 con 0x7f198406ded0 2024-04-25T19:12:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.148+0000 7f198c227640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f1984163c30 con 0x7f198415f7d0 2024-04-25T19:12:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.148+0000 7f198c227640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f1984164a90 con 0x7f19840b5130 2024-04-25T19:12:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f198a7c4640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f198415f7d0 0x7f198415fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f19897c2640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f198406ded0 0x7f198406e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39692/0 (socket says 172.21.15.28:39692) 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f19897c2640 1 -- 172.21.15.28:0/3743141603 learned_addr learned my addr 172.21.15.28:0/3743141603 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f1989fc3640 1 --2- 172.21.15.28:0/3743141603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b5130 0x7f1984161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f198a7c4640 1 -- 172.21.15.28:0/3743141603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b5130 msgr2=0x7f1984161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f198a7c4640 1 --2- 172.21.15.28:0/3743141603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b5130 0x7f1984161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f198a7c4640 1 -- 172.21.15.28:0/3743141603 >> v1:172.21.15.28:6789/0 conn(0x7f198406ded0 legacy=0x7f198406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f1988fc1640 1 -- 172.21.15.28:0/3743141603 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2256059568 0 0) 0x7f1984163e80 con 0x7f198406ded0 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f198a7c4640 1 -- 172.21.15.28:0/3743141603 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1984164160 con 0x7f198415f7d0 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.149+0000 7f1989fc3640 1 --2- 172.21.15.28:0/3743141603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b5130 0x7f1984161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f198a7c4640 1 --2- 172.21.15.28:0/3743141603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198415f7d0 0x7f198415fbe0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f197c009d90 tx=0x7f197c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6823eb1bfbb2fd29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f1988fc1640 1 -- 172.21.15.28:0/3743141603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c007740 con 0x7f198415f7d0 2024-04-25T19:12:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f1988fc1640 1 -- 172.21.15.28:0/3743141603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f197c0078e0 con 0x7f198415f7d0 2024-04-25T19:12:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f1988fc1640 1 -- 172.21.15.28:0/3743141603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197c007cb0 con 0x7f198415f7d0 2024-04-25T19:12:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f198c227640 1 -- 172.21.15.28:0/3743141603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198415f7d0 msgr2=0x7f198415fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f198c227640 1 --2- 172.21.15.28:0/3743141603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198415f7d0 0x7f198415fbe0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f197c009d90 tx=0x7f197c02a660 comp rx=0 tx=0).stop 2024-04-25T19:12:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f198c227640 1 -- 172.21.15.28:0/3743141603 shutdown_connections 2024-04-25T19:12:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f198c227640 1 --2- 172.21.15.28:0/3743141603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198415f7d0 0x7f198415fbe0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f198c227640 1 --2- 172.21.15.28:0/3743141603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f19840b5130 0x7f1984161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.150+0000 7f198c227640 1 -- 172.21.15.28:0/3743141603 >> 172.21.15.28:0/3743141603 conn(0x7f19840b7ad0 msgr2=0x7f19840b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.151+0000 7f198c227640 1 -- 172.21.15.28:0/3743141603 shutdown_connections 2024-04-25T19:12:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.151+0000 7f198c227640 1 -- 172.21.15.28:0/3743141603 wait complete. 2024-04-25T19:12:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.152+0000 7f198c227640 1 Processor -- start 2024-04-25T19:12:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.152+0000 7f198c227640 1 -- start start 2024-04-25T19:12:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198c227640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198406ded0 0x7f1984159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198c227640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19840b5130 0x7f1984157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198c227640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198415f7d0 0x7f19841622b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198c227640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1984160120 con 0x7f198406ded0 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198c227640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f1984164720 con 0x7f19840b5130 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198c227640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f19841602a0 con 0x7f198415f7d0 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198406ded0 0x7f1984159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198406ded0 0x7f1984159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54040/0 (socket says 172.21.15.28:54040) 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 -- 172.21.15.28:0/4008843025 learned_addr learned my addr 172.21.15.28:0/4008843025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f19897c2640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19840b5130 0x7f1984157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198a7c4640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198415f7d0 0x7f19841622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 -- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198415f7d0 msgr2=0x7f19841622b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198415f7d0 0x7f19841622b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 -- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19840b5130 msgr2=0x7f1984157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19840b5130 0x7f1984157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 -- 172.21.15.28:0/4008843025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f198415a630 con 0x7f198406ded0 2024-04-25T19:12:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f19897c2640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19840b5130 0x7f1984157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f1989fc3640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198406ded0 0x7f1984159cc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f198000d7d0 tx=0x7f198000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f197a7fc640 1 -- 172.21.15.28:0/4008843025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19800096a0 con 0x7f198406ded0 2024-04-25T19:12:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f197a7fc640 1 -- 172.21.15.28:0/4008843025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1980019070 con 0x7f198406ded0 2024-04-25T19:12:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f198415a910 con 0x7f198406ded0 2024-04-25T19:12:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f197a7fc640 1 -- 172.21.15.28:0/4008843025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1980009a80 con 0x7f198406ded0 2024-04-25T19:12:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.154+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19841ea710 con 0x7f198406ded0 2024-04-25T19:12:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.155+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1984050740 con 0x7f198406ded0 2024-04-25T19:12:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.156+0000 7f197a7fc640 1 -- 172.21.15.28:0/4008843025 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1980009c20 con 0x7f198406ded0 2024-04-25T19:12:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.157+0000 7f197a7fc640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19680744e0 0x7f19680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.157+0000 7f197a7fc640 1 -- 172.21.15.28:0/4008843025 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f19800956d0 con 0x7f198406ded0 2024-04-25T19:12:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.157+0000 7f19897c2640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19680744e0 0x7f19680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.157+0000 7f19897c2640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19680744e0 0x7f19680769f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f198415d2c0 tx=0x7f1974009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:45.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.161+0000 7f197a7fc640 1 -- 172.21.15.28:0/4008843025 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f198005fdf0 con 0x7f198406ded0 2024-04-25T19:12:45.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.346+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19840d2fe0 con 0x7f198406ded0 2024-04-25T19:12:45.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.347+0000 7f197a7fc640 1 -- 172.21.15.28:0/4008843025 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16895 (secure 0 0 0) 0x7f198005f6f0 con 0x7f198406ded0 2024-04-25T19:12:45.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:45.349 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:44.063334+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19680744e0 msgr2=0x7f19680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19680744e0 0x7f19680769f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f198415d2c0 tx=0x7f1974009040 comp rx=0 tx=0).stop 2024-04-25T19:12:45.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198406ded0 msgr2=0x7f1984159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198406ded0 0x7f1984159cc0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f198000d7d0 tx=0x7f198000dc90 comp rx=0 tx=0).stop 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 shutdown_connections 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19680744e0 0x7f19680769f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f198415f7d0 0x7f19841622b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19840b5130 0x7f1984157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 --2- 172.21.15.28:0/4008843025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f198406ded0 0x7f1984159cc0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 >> 172.21.15.28:0/4008843025 conn(0x7f19840b7ad0 msgr2=0x7f19840d3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 shutdown_connections 2024-04-25T19:12:45.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.349+0000 7f198c227640 1 -- 172.21.15.28:0/4008843025 wait complete. 2024-04-25T19:12:45.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T19:12:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.477+0000 7f5b905c5640 1 Processor -- start 2024-04-25T19:12:45.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.477+0000 7f5b905c5640 1 -- start start 2024-04-25T19:12:45.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.478+0000 7f5b905c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b880d2d10 0x7f5b880d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.478+0000 7f5b905c5640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b880b3d00 0x7f5b880b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b905c5640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b880b92a0 con 0x7f5b880b4650 2024-04-25T19:12:45.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b905c5640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5b8809cea0 con 0x7f5b880b3d00 2024-04-25T19:12:45.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b905c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5b8806e8e0 con 0x7f5b880d2d10 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8eb62640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5b880b4650 0x7f5b8806d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39696/0 (socket says 172.21.15.28:39696) 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8eb62640 1 -- 172.21.15.28:0/3282073712 learned_addr learned my addr 172.21.15.28:0/3282073712 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8e361640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b880d2d10 0x7f5b880d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8db60640 1 --2- 172.21.15.28:0/3282073712 >> v2:172.21.15.53:3300/0 conn(0x7f5b880b3d00 0x7f5b880b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 821443833 0 0) 0x7f5b880b92a0 con 0x7f5b880b4650 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5b70003520 con 0x7f5b880b4650 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3286013691 0 0) 0x7f5b70003520 con 0x7f5b880b4650 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b880d2d10 msgr2=0x7f5b880d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b880d2d10 0x7f5b880d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 >> v2:172.21.15.53:3300/0 conn(0x7f5b880b3d00 msgr2=0x7f5b880b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 --2- 172.21.15.28:0/3282073712 >> v2:172.21.15.53:3300/0 conn(0x7f5b880b3d00 0x7f5b880b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b880b8ee0 con 0x7f5b880b4650 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f5b78001670 con 0x7f5b880b4650 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 >> v1:172.21.15.28:6789/0 conn(0x7f5b880b4650 legacy=0x7f5b8806d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b70003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b880d2d10 0x7f5b70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b70004d40 0x7f5b70005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8809cea0 con 0x7f5b700039b0 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8806e8e0 con 0x7f5b70004d40 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5b7000ba40 con 0x7f5b880d2d10 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.479+0000 7f5b8e361640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b70003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.480+0000 7f5b8db60640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b880d2d10 0x7f5b70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.480+0000 7f5b8eb62640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b70004d40 0x7f5b70005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.480+0000 7f5b8e361640 1 -- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b880d2d10 msgr2=0x7f5b70004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.480+0000 7f5b8e361640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b880d2d10 0x7f5b70004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.480+0000 7f5b8e361640 1 -- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b70004d40 msgr2=0x7f5b70005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.480+0000 7f5b8e361640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b70004d40 0x7f5b70005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.480+0000 7f5b8e361640 1 -- 172.21.15.28:0/3282073712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b70003520 con 0x7f5b700039b0 2024-04-25T19:12:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b8e361640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b70003e20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009f90 tx=0x7f5b70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de74d37695fe7a13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80009c10 con 0x7f5b700039b0 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b80036070 con 0x7f5b700039b0 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b8d35f640 1 -- 172.21.15.28:0/3282073712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80031810 con 0x7f5b700039b0 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b905c5640 1 -- 172.21.15.28:0/3282073712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 msgr2=0x7f5b70003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b905c5640 1 --2- 172.21.15.28:0/3282073712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b70003e20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009f90 tx=0x7f5b70005830 comp rx=0 tx=0).stop 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b8eb62640 1 -- 172.21.15.28:0/3282073712 reap_dead start 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b905c5640 1 -- 172.21.15.28:0/3282073712 shutdown_connections 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.481+0000 7f5b905c5640 1 -- 172.21.15.28:0/3282073712 >> 172.21.15.28:0/3282073712 conn(0x7f5b880d0450 msgr2=0x7f5b880d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.482+0000 7f5b905c5640 1 -- 172.21.15.28:0/3282073712 shutdown_connections 2024-04-25T19:12:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.482+0000 7f5b905c5640 1 -- 172.21.15.28:0/3282073712 wait complete. 2024-04-25T19:12:45.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.483+0000 7f5b905c5640 1 Processor -- start 2024-04-25T19:12:45.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.483+0000 7f5b905c5640 1 -- start start 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b905c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b881f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b905c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b881dbbb0 0x7f5b881eb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b905c5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b881eb690 0x7f5b880caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b905c5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5b880d3600 con 0x7f5b700039b0 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b905c5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5b880d3120 con 0x7f5b881eb690 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b905c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5b880d3780 con 0x7f5b881dbbb0 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b881f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b881f6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54056/0 (socket says 172.21.15.28:54056) 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 -- 172.21.15.28:0/621758504 learned_addr learned my addr 172.21.15.28:0/621758504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8eb62640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b881eb690 0x7f5b880caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8db60640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b881dbbb0 0x7f5b881eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 -- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b881dbbb0 msgr2=0x7f5b881eb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b881dbbb0 0x7f5b881eb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 -- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b881eb690 msgr2=0x7f5b880caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b881eb690 0x7f5b880caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 -- 172.21.15.28:0/621758504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b881f7020 con 0x7f5b700039b0 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8db60640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b881dbbb0 0x7f5b881eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b8e361640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b881f6ae0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009f90 tx=0x7f5b80009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b86ffd640 1 -- 172.21.15.28:0/621758504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80009c10 con 0x7f5b700039b0 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b86ffd640 1 -- 172.21.15.28:0/621758504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b80045630 con 0x7f5b700039b0 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.485+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b881ebaa0 con 0x7f5b700039b0 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.486+0000 7f5b86ffd640 1 -- 172.21.15.28:0/621758504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80045a20 con 0x7f5b700039b0 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.486+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b881e9ac0 con 0x7f5b700039b0 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.487+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b8806f090 con 0x7f5b700039b0 2024-04-25T19:12:45.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.487+0000 7f5b86ffd640 1 -- 172.21.15.28:0/621758504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5b80005d10 con 0x7f5b700039b0 2024-04-25T19:12:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.488+0000 7f5b86ffd640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5b740744b0 0x7f5b740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.488+0000 7f5b86ffd640 1 -- 172.21.15.28:0/621758504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f5b800b70e0 con 0x7f5b700039b0 2024-04-25T19:12:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.488+0000 7f5b8db60640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5b740744b0 0x7f5b740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:45.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.488+0000 7f5b8db60640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5b740744b0 0x7f5b740769c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5b881f3c50 tx=0x7f5b7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:45.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.492+0000 7f5b86ffd640 1 -- 172.21.15.28:0/621758504 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5b8806f090 con 0x7f5b700039b0 2024-04-25T19:12:45.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:45.727+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f5b88050740 con 0x7f5b700039b0 2024-04-25T19:12:46.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.694+0000 7f5b86ffd640 1 -- 172.21.15.28:0/621758504 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f5b881ebaa0 con 0x7f5b700039b0 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.697+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5b740744b0 msgr2=0x7f5b740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.697+0000 7f5b905c5640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5b740744b0 0x7f5b740769c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5b881f3c50 tx=0x7f5b7c008040 comp rx=0 tx=0).stop 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.697+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 msgr2=0x7f5b881f6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.697+0000 7f5b905c5640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b881f6ae0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009f90 tx=0x7f5b80009750 comp rx=0 tx=0).stop 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.698+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 shutdown_connections 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.698+0000 7f5b905c5640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5b740744b0 0x7f5b740769c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.698+0000 7f5b905c5640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b881eb690 0x7f5b880caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.698+0000 7f5b905c5640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5b881dbbb0 0x7f5b881eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.698+0000 7f5b905c5640 1 --2- 172.21.15.28:0/621758504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b700039b0 0x7f5b881f6ae0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.698+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 >> 172.21.15.28:0/621758504 conn(0x7f5b880d0450 msgr2=0x7f5b880d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:46.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.698+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 shutdown_connections 2024-04-25T19:12:46.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.698+0000 7f5b905c5640 1 -- 172.21.15.28:0/621758504 wait complete. 2024-04-25T19:12:46.713 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T19:12:46.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T19:12:46.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.833+0000 7f5fbea82640 1 Processor -- start 2024-04-25T19:12:46.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.833+0000 7f5fbea82640 1 -- start start 2024-04-25T19:12:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.834+0000 7f5fbea82640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5fb8160e00 0x7f5fb815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:46.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.834+0000 7f5fbea82640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb8161210 con 0x7f5fb8164990 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.834+0000 7f5fbea82640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fb81614a0 con 0x7f5fb80b7d70 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.834+0000 7f5fbea82640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5fb8163eb0 con 0x7f5fb8160e00 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fbc81e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5fb8164990 0x7f5fb80b7660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39706/0 (socket says 172.21.15.28:39706) 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fbc81e640 1 -- 172.21.15.28:0/2965003785 learned_addr learned my addr 172.21.15.28:0/2965003785 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750736174 0 0) 0x7f5fb8161210 con 0x7f5fb8164990 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fa8003520 con 0x7f5fb8164990 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fbd01f640 1 --2- 172.21.15.28:0/2965003785 >> v2:172.21.15.53:3300/0 conn(0x7f5fb8160e00 0x7f5fb815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 600125298 0 0) 0x7f5fa8003520 con 0x7f5fb8164990 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 >> v2:172.21.15.53:3300/0 conn(0x7f5fb8160e00 msgr2=0x7f5fb815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 --2- 172.21.15.28:0/2965003785 >> v2:172.21.15.53:3300/0 conn(0x7f5fb8160e00 0x7f5fb815dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 >> v1:172.21.15.53:6789/0 conn(0x7f5fb80b7d70 legacy=0x7f5fb81606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:46.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb8164190 con 0x7f5fb8164990 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f5fac001670 con 0x7f5fb8164990 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 >> v1:172.21.15.28:6789/0 conn(0x7f5fb8164990 legacy=0x7f5fb80b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa80039b0 0x7f5fa8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa8004800 0x7f5fa8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fa80051c0 0x7f5fa800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb8163eb0 con 0x7f5fa80039b0 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa800c160 con 0x7f5fa8004800 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa800c360 con 0x7f5fa80051c0 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fbc81e640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa80039b0 0x7f5fa8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fb7fff640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa8004800 0x7f5fa8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.835+0000 7f5fbd01f640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fa80051c0 0x7f5fa800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:46.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.836+0000 7f5fbc81e640 1 -- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fa80051c0 msgr2=0x7f5fa800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.836+0000 7f5fbc81e640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fa80051c0 0x7f5fa800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.836+0000 7f5fbc81e640 1 -- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa8004800 msgr2=0x7f5fa8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.836+0000 7f5fbc81e640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa8004800 0x7f5fa8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.836+0000 7f5fbc81e640 1 -- 172.21.15.28:0/2965003785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa8003520 con 0x7f5fa80039b0 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.836+0000 7f5fbd01f640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fa80051c0 0x7f5fa800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.836+0000 7f5fbd01f640 1 -- 172.21.15.28:0/2965003785 reap_dead start 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.836+0000 7f5fbc81e640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa80039b0 0x7f5fa8003e20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00c210 tx=0x7f5fa8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765b07fecb698e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac007cf0 con 0x7f5fa80039b0 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fac005af0 con 0x7f5fa80039b0 2024-04-25T19:12:46.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fb77fe640 1 -- 172.21.15.28:0/2965003785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fac007070 con 0x7f5fa80039b0 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fbea82640 1 -- 172.21.15.28:0/2965003785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa80039b0 msgr2=0x7f5fa8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fbea82640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa80039b0 0x7f5fa8003e20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5fac00c210 tx=0x7f5fa8005830 comp rx=0 tx=0).stop 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fbea82640 1 -- 172.21.15.28:0/2965003785 shutdown_connections 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fbea82640 1 --2- 172.21.15.28:0/2965003785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fa80039b0 0x7f5fa8003e20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fbea82640 1 -- 172.21.15.28:0/2965003785 >> 172.21.15.28:0/2965003785 conn(0x7f5fb80d2840 msgr2=0x7f5fb80d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fbea82640 1 -- 172.21.15.28:0/2965003785 shutdown_connections 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.837+0000 7f5fbea82640 1 -- 172.21.15.28:0/2965003785 wait complete. 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.839+0000 7f5fbea82640 1 Processor -- start 2024-04-25T19:12:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.839+0000 7f5fbea82640 1 -- start start 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbea82640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa80039b0 0x7f5fb815d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbea82640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fb81e9ab0 0x7f5fb80ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbea82640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb81fa8a0 0x7f5fb81facb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbea82640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb815de30 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbea82640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5fb815dcb0 con 0x7f5fa80039b0 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbea82640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5fb815dfb0 con 0x7f5fb81e9ab0 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb81fa8a0 0x7f5fb81facb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb81fa8a0 0x7f5fb81facb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54060/0 (socket says 172.21.15.28:54060) 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 -- 172.21.15.28:0/2271755296 learned_addr learned my addr 172.21.15.28:0/2271755296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbc81e640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa80039b0 0x7f5fb815d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fb7fff640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fb81e9ab0 0x7f5fb80ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 -- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fb81e9ab0 msgr2=0x7f5fb80ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:46.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fb81e9ab0 0x7f5fb80ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 -- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa80039b0 msgr2=0x7f5fb815d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa80039b0 0x7f5fb815d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 -- 172.21.15.28:0/2271755296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fb815d5a0 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fb7fff640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fb81e9ab0 0x7f5fb80ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fbd01f640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb81fa8a0 0x7f5fb81facb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80d2440 tx=0x7f5fb0010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.841+0000 7f5fb57fa640 1 -- 172.21.15.28:0/2271755296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0010e60 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.842+0000 7f5fb57fa640 1 -- 172.21.15.28:0/2271755296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fb0003990 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.842+0000 7f5fb57fa640 1 -- 172.21.15.28:0/2271755296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb0003da0 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.842+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fb81fb320 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.842+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fb80c9890 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.843+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fb81fb840 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.843+0000 7f5fb57fa640 1 -- 172.21.15.28:0/2271755296 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5fb001e020 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.844+0000 7f5fb57fa640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5f9c0744e0 0x7f5f9c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.844+0000 7f5fbc81e640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5f9c0744e0 0x7f5f9c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.844+0000 7f5fb57fa640 1 -- 172.21.15.28:0/2271755296 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f5fb00959d0 con 0x7f5fb81fa8a0 2024-04-25T19:12:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.845+0000 7f5fbc81e640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5f9c0744e0 0x7f5f9c0769f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5fac032000 tx=0x7f5fac009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:46.849+0000 7f5fb57fa640 1 -- 172.21.15.28:0/2271755296 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5fb005ffd0 con 0x7f5fb81fa8a0 2024-04-25T19:12:47.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.074+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5fb8050740 con 0x7f5fb81fa8a0 2024-04-25T19:12:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.695+0000 7f5fb57fa640 1 -- 172.21.15.28:0/2271755296 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) ==== 122+0+0 (secure 0 0 0) 0x7f5fb005f8d0 con 0x7f5fb81fa8a0 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.697+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5f9c0744e0 msgr2=0x7f5f9c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5f9c0744e0 0x7f5f9c0769f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5fac032000 tx=0x7f5fac009030 comp rx=0 tx=0).stop 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb81fa8a0 msgr2=0x7f5fb81facb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb81fa8a0 0x7f5fb81facb0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5fb80d2440 tx=0x7f5fb0010590 comp rx=0 tx=0).stop 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 shutdown_connections 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5f9c0744e0 0x7f5f9c0769f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5fb81fa8a0 0x7f5fb81facb0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5fb81e9ab0 0x7f5fb80ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 --2- 172.21.15.28:0/2271755296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fa80039b0 0x7f5fb815d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 >> 172.21.15.28:0/2271755296 conn(0x7f5fb80d2840 msgr2=0x7f5fb815fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 shutdown_connections 2024-04-25T19:12:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.698+0000 7f5fbea82640 1 -- 172.21.15.28:0/2271755296 wait complete. 2024-04-25T19:12:47.714 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T19:12:47.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.830+0000 7f537c314640 1 Processor -- start 2024-04-25T19:12:47.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.830+0000 7f537c314640 1 -- start start 2024-04-25T19:12:47.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537c314640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53740b4e30 0x7f53740bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:47.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537c314640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53740b9470 0x7f53740b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:47.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537c314640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53740bbb90 con 0x7f53740b7650 2024-04-25T19:12:47.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537c314640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f53740d0b50 con 0x7f53740b9470 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537c314640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f537408e3a0 con 0x7f53740b4e30 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537a8b1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f53740b7650 0x7f53740b7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39714/0 (socket says 172.21.15.28:39714) 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f53798af640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53740b9470 0x7f53740b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537a8b1640 1 -- 172.21.15.28:0/2132779473 learned_addr learned my addr 172.21.15.28:0/2132779473 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537a0b0640 1 --2- 172.21.15.28:0/2132779473 >> v2:172.21.15.53:3300/0 conn(0x7f53740b4e30 0x7f53740bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f53798af640 1 -- 172.21.15.28:0/2132779473 >> v2:172.21.15.53:3300/0 conn(0x7f53740b4e30 msgr2=0x7f53740bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f53798af640 1 --2- 172.21.15.28:0/2132779473 >> v2:172.21.15.53:3300/0 conn(0x7f53740b4e30 0x7f53740bb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f53798af640 1 -- 172.21.15.28:0/2132779473 >> v1:172.21.15.28:6789/0 conn(0x7f53740b7650 legacy=0x7f53740b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f53798af640 1 -- 172.21.15.28:0/2132779473 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5374164d60 con 0x7f53740b9470 2024-04-25T19:12:47.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.832+0000 7f537a0b0640 1 --2- 172.21.15.28:0/2132779473 >> v2:172.21.15.53:3300/0 conn(0x7f53740b4e30 0x7f53740bb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:47.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.833+0000 7f53798af640 1 --2- 172.21.15.28:0/2132779473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b9470 0x7f53740b9880 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f537408e8e0 tx=0x7f536802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945b6af6aa2b7279 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.833+0000 7f53790ae640 1 -- 172.21.15.28:0/2132779473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368037070 con 0x7f53740b9470 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.833+0000 7f53790ae640 1 -- 172.21.15.28:0/2132779473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5368032440 con 0x7f53740b9470 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.833+0000 7f53790ae640 1 -- 172.21.15.28:0/2132779473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536802ae60 con 0x7f53740b9470 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.834+0000 7f537c314640 1 -- 172.21.15.28:0/2132779473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b9470 msgr2=0x7f53740b9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.834+0000 7f537c314640 1 --2- 172.21.15.28:0/2132779473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b9470 0x7f53740b9880 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f537408e8e0 tx=0x7f536802a110 comp rx=0 tx=0).stop 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.834+0000 7f537c314640 1 -- 172.21.15.28:0/2132779473 shutdown_connections 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.834+0000 7f537c314640 1 --2- 172.21.15.28:0/2132779473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b9470 0x7f53740b9880 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.834+0000 7f537c314640 1 --2- 172.21.15.28:0/2132779473 >> v2:172.21.15.53:3300/0 conn(0x7f53740b4e30 0x7f53740bb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.834+0000 7f537c314640 1 -- 172.21.15.28:0/2132779473 >> 172.21.15.28:0/2132779473 conn(0x7f53740d2650 msgr2=0x7f53740d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.834+0000 7f537c314640 1 -- 172.21.15.28:0/2132779473 shutdown_connections 2024-04-25T19:12:47.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.834+0000 7f537c314640 1 -- 172.21.15.28:0/2132779473 wait complete. 2024-04-25T19:12:47.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.836+0000 7f537c314640 1 Processor -- start 2024-04-25T19:12:47.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.836+0000 7f537c314640 1 -- start start 2024-04-25T19:12:47.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f537c314640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53740b4e30 0x7f5374156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f537c314640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b7650 0x7f53740ba410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f537c314640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53740b9470 0x7f53740ba950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f537c314640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53740d07e0 con 0x7f53740b7650 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f537c314640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f53740d0500 con 0x7f53740b4e30 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f537c314640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f53741649c0 con 0x7f53740b9470 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f53798af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b7650 0x7f53740ba410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f53798af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b7650 0x7f53740ba410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54084/0 (socket says 172.21.15.28:54084) 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.837+0000 7f53798af640 1 -- 172.21.15.28:0/2863639897 learned_addr learned my addr 172.21.15.28:0/2863639897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f537a0b0640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53740b4e30 0x7f5374156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f537a8b1640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53740b9470 0x7f53740ba950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f53798af640 1 -- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53740b9470 msgr2=0x7f53740ba950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f53798af640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53740b9470 0x7f53740ba950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f53798af640 1 -- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53740b4e30 msgr2=0x7f5374156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f53798af640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53740b4e30 0x7f5374156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f53798af640 1 -- 172.21.15.28:0/2863639897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53741faae0 con 0x7f53740b7650 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f537a0b0640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53740b4e30 0x7f5374156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f53798af640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b7650 0x7f53740ba410 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5368000c00 tx=0x7f53680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f5372ffd640 1 -- 172.21.15.28:0/2863639897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368037040 con 0x7f53740b7650 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f5372ffd640 1 -- 172.21.15.28:0/2863639897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5368042070 con 0x7f53740b7650 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f537415d4f0 con 0x7f53740b7650 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f5372ffd640 1 -- 172.21.15.28:0/2863639897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368030bd0 con 0x7f53740b7650 2024-04-25T19:12:47.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.838+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53741fa800 con 0x7f53740b7650 2024-04-25T19:12:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.839+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5374050740 con 0x7f53740b7650 2024-04-25T19:12:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.840+0000 7f5372ffd640 1 -- 172.21.15.28:0/2863639897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5368047460 con 0x7f53740b7650 2024-04-25T19:12:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.840+0000 7f5372ffd640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53600744b0 0x7f53600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.841+0000 7f537a0b0640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53600744b0 0x7f53600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.841+0000 7f5372ffd640 1 -- 172.21.15.28:0/2863639897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f53680b5f30 con 0x7f53740b7650 2024-04-25T19:12:47.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.841+0000 7f537a0b0640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53600744b0 0x7f53600769c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f536c006fd0 tx=0x7f536c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:47.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:47.845+0000 7f5372ffd640 1 -- 172.21.15.28:0/2863639897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5368032e20 con 0x7f53740b7650 2024-04-25T19:12:48.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.067+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53740b83a0 con 0x7f53740b7650 2024-04-25T19:12:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.068+0000 7f5372ffd640 1 -- 172.21.15.28:0/2863639897 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5169 (secure 0 0 0) 0x7f53680805c0 con 0x7f53740b7650 2024-04-25T19:12:48.070 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:48.070 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/2695669357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2695669357},{"type":"v1","addr":"172.21.15.53:6834","nonce":2695669357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/2737682189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2737682189},{"type":"v1","addr":"172.21.15.28:6836","nonce":2737682189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/6663801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":6663801},{"type":"v1","addr":"172.21.15.28:6837","nonce":6663801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:47.076286+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/803470605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":803470605},{"type":"v1","addr":"172.21.15.53:6835","nonce":803470605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":2}]} 2024-04-25T19:12:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-04-25T19:12:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53600744b0 msgr2=0x7f53600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53600744b0 0x7f53600769c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f536c006fd0 tx=0x7f536c008040 comp rx=0 tx=0).stop 2024-04-25T19:12:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b7650 msgr2=0x7f53740ba410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b7650 0x7f53740ba410 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5368000c00 tx=0x7f53680046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 shutdown_connections 2024-04-25T19:12:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53600744b0 0x7f53600769c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53740b9470 0x7f53740ba950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53740b7650 0x7f53740ba410 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 --2- 172.21.15.28:0/2863639897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53740b4e30 0x7f5374156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 >> 172.21.15.28:0/2863639897 conn(0x7f53740d2650 msgr2=0x7f5374160000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 shutdown_connections 2024-04-25T19:12:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.070+0000 7f537c314640 1 -- 172.21.15.28:0/2863639897 wait complete. 2024-04-25T19:12:48.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.192+0000 7fa9cdf46640 1 Processor -- start 2024-04-25T19:12:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.192+0000 7fa9cdf46640 1 -- start start 2024-04-25T19:12:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9cdf46640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9c8164990 0x7fa9c815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9cdf46640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa9c81606f0 0x7fa9c8160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9cdf46640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9c81614a0 con 0x7fa9c80b7660 2024-04-25T19:12:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9cdf46640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fa9c8163eb0 con 0x7fa9c8164990 2024-04-25T19:12:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9cdf46640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fa9c8164190 con 0x7fa9c81606f0 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c77fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9c8164990 0x7fa9c815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c6ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa9c80b7660 0x7fa9c80b7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39730/0 (socket says 172.21.15.28:39730) 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c6ffd640 1 -- 172.21.15.28:0/2754442522 learned_addr learned my addr 172.21.15.28:0/2754442522 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c7fff640 1 --2- 172.21.15.28:0/2754442522 >> v2:172.21.15.53:3300/0 conn(0x7fa9c81606f0 0x7fa9c8160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c77fe640 1 -- 172.21.15.28:0/2754442522 >> v2:172.21.15.53:3300/0 conn(0x7fa9c81606f0 msgr2=0x7fa9c8160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c77fe640 1 --2- 172.21.15.28:0/2754442522 >> v2:172.21.15.53:3300/0 conn(0x7fa9c81606f0 0x7fa9c8160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c77fe640 1 -- 172.21.15.28:0/2754442522 >> v1:172.21.15.28:6789/0 conn(0x7fa9c80b7660 legacy=0x7fa9c80b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c67fc640 1 -- 172.21.15.28:0/2754442522 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323190206 0 0) 0x7fa9c81614a0 con 0x7fa9c80b7660 2024-04-25T19:12:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.194+0000 7fa9c77fe640 1 -- 172.21.15.28:0/2754442522 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c8164470 con 0x7fa9c8164990 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.195+0000 7fa9c77fe640 1 --2- 172.21.15.28:0/2754442522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c8164990 0x7fa9c815dcb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80c96e0 tx=0x7fa9b802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715b7f7d16b18da7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.195+0000 7fa9c67fc640 1 -- 172.21.15.28:0/2754442522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8037070 con 0x7fa9c8164990 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.195+0000 7fa9c67fc640 1 -- 172.21.15.28:0/2754442522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b8032440 con 0x7fa9c8164990 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.195+0000 7fa9c67fc640 1 -- 172.21.15.28:0/2754442522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8032810 con 0x7fa9c8164990 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.196+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2754442522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c8164990 msgr2=0x7fa9c815dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.196+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2754442522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c8164990 0x7fa9c815dcb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80c96e0 tx=0x7fa9b802a1a0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.196+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2754442522 shutdown_connections 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.196+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2754442522 >> v2:172.21.15.53:3300/0 conn(0x7fa9c81606f0 0x7fa9c8160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.196+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2754442522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c8164990 0x7fa9c815dcb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.196+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2754442522 >> 172.21.15.28:0/2754442522 conn(0x7fa9c80d2840 msgr2=0x7fa9c80d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.196+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2754442522 shutdown_connections 2024-04-25T19:12:48.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.196+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2754442522 wait complete. 2024-04-25T19:12:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.198+0000 7fa9cdf46640 1 Processor -- start 2024-04-25T19:12:48.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.198+0000 7fa9cdf46640 1 -- start start 2024-04-25T19:12:48.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.199+0000 7fa9cdf46640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c80b7660 0x7fa9c815a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.199+0000 7fa9cdf46640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9c81606f0 0x7fa9c80c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.199+0000 7fa9cdf46640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c8164990 0x7fa9c80c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.199+0000 7fa9cdf46640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c80b4750 con 0x7fa9c80b7660 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.199+0000 7fa9cdf46640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c80b45d0 con 0x7fa9c8164990 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.199+0000 7fa9cdf46640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa9c80b48d0 con 0x7fa9c81606f0 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c80b7660 0x7fa9c815a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c80b7660 0x7fa9c815a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54096/0 (socket says 172.21.15.28:54096) 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 -- 172.21.15.28:0/2178133928 learned_addr learned my addr 172.21.15.28:0/2178133928 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c7fff640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c8164990 0x7fa9c80c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c6ffd640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9c81606f0 0x7fa9c80c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 -- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9c81606f0 msgr2=0x7fa9c80c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9c81606f0 0x7fa9c80c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 -- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c8164990 msgr2=0x7fa9c80c9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c8164990 0x7fa9c80c9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 -- 172.21.15.28:0/2178133928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c815a990 con 0x7fa9c80b7660 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c7fff640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c8164990 0x7fa9c80c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c6ffd640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9c81606f0 0x7fa9c80c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9c77fe640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c80b7660 0x7fa9c815a450 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009890 tx=0x7fa9b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9affff640 1 -- 172.21.15.28:0/2178133928 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8037040 con 0x7fa9c80b7660 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9affff640 1 -- 172.21.15.28:0/2178133928 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b8042070 con 0x7fa9c80b7660 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c815ea80 con 0x7fa9c80b7660 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9affff640 1 -- 172.21.15.28:0/2178133928 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b8030920 con 0x7fa9c80b7660 2024-04-25T19:12:48.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.200+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c815f040 con 0x7fa9c80b7660 2024-04-25T19:12:48.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.201+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c8162f30 con 0x7fa9c80b7660 2024-04-25T19:12:48.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.202+0000 7fa9affff640 1 -- 172.21.15.28:0/2178133928 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa9b8030ac0 con 0x7fa9c80b7660 2024-04-25T19:12:48.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.202+0000 7fa9affff640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9b40744b0 0x7fa9b40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.203+0000 7fa9affff640 1 -- 172.21.15.28:0/2178133928 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fa9b80b5c30 con 0x7fa9c80b7660 2024-04-25T19:12:48.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.203+0000 7fa9c6ffd640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9b40744b0 0x7fa9b40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.203+0000 7fa9c6ffd640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9b40744b0 0x7fa9b40769c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80d12c0 tx=0x7fa9bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.207+0000 7fa9affff640 1 -- 172.21.15.28:0/2178133928 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9c8162f30 con 0x7fa9c80b7660 2024-04-25T19:12:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.391+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9c8050740 con 0x7fa9c80b7660 2024-04-25T19:12:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.392+0000 7fa9affff640 1 -- 172.21.15.28:0/2178133928 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16882 (secure 0 0 0) 0x7fa9b804c020 con 0x7fa9c80b7660 2024-04-25T19:12:48.394 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:48.394 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":38,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:45.682011+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.394+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9b40744b0 msgr2=0x7fa9b40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.394+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9b40744b0 0x7fa9b40769c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa9c80d12c0 tx=0x7fa9bc009040 comp rx=0 tx=0).stop 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.394+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c80b7660 msgr2=0x7fa9c815a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.394+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c80b7660 0x7fa9c815a450 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fa9b8009890 tx=0x7fa9b80046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.395+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 shutdown_connections 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.395+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9b40744b0 0x7fa9b40769c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.395+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c8164990 0x7fa9c80c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.395+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9c81606f0 0x7fa9c80c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.395+0000 7fa9cdf46640 1 --2- 172.21.15.28:0/2178133928 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9c80b7660 0x7fa9c815a450 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.395+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 >> 172.21.15.28:0/2178133928 conn(0x7fa9c80d2840 msgr2=0x7fa9c80b7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.395+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 shutdown_connections 2024-04-25T19:12:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.395+0000 7fa9cdf46640 1 -- 172.21.15.28:0/2178133928 wait complete. 2024-04-25T19:12:48.408 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T19:12:48.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:12:48.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.518+0000 7fbc3ba34640 1 Processor -- start 2024-04-25T19:12:48.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.518+0000 7fbc3ba34640 1 -- start start 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.519+0000 7fbc3ba34640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc340b5130 0x7fbc34161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.519+0000 7fbc3ba34640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3406ded0 0x7fbc3406e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.519+0000 7fbc3ba34640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc3406e820 con 0x7fbc340d0830 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.519+0000 7fbc3ba34640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fbc34163c30 con 0x7fbc340b5130 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.519+0000 7fbc3ba34640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbc34164a90 con 0x7fbc3406ded0 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.519+0000 7fbc397d0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbc340b5130 0x7fbc34161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.519+0000 7fbc39fd1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbc340d0830 0x7fbc340d0c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39732/0 (socket says 172.21.15.28:39732) 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.519+0000 7fbc39fd1640 1 -- 172.21.15.28:0/2042453754 learned_addr learned my addr 172.21.15.28:0/2042453754 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.520+0000 7fbc38fcf640 1 --2- 172.21.15.28:0/2042453754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3406ded0 0x7fbc3406e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.520+0000 7fbc2bfff640 1 -- 172.21.15.28:0/2042453754 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3630627076 0 0) 0x7fbc2c001490 con 0x7fbc340d0830 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.520+0000 7fbc2bfff640 1 -- 172.21.15.28:0/2042453754 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc20003520 con 0x7fbc340d0830 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.520+0000 7fbc397d0640 1 -- 172.21.15.28:0/2042453754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3406ded0 msgr2=0x7fbc3406e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.520+0000 7fbc397d0640 1 --2- 172.21.15.28:0/2042453754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3406ded0 0x7fbc3406e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.520+0000 7fbc397d0640 1 -- 172.21.15.28:0/2042453754 >> v1:172.21.15.28:6789/0 conn(0x7fbc340d0830 legacy=0x7fbc340d0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.520+0000 7fbc397d0640 1 -- 172.21.15.28:0/2042453754 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc34163e80 con 0x7fbc340b5130 2024-04-25T19:12:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc397d0640 1 --2- 172.21.15.28:0/2042453754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc340b5130 0x7fbc34161130 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbc240097a0 tx=0x7fbc2402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afa4273fc5940ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc2bfff640 1 -- 172.21.15.28:0/2042453754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24037070 con 0x7fbc340b5130 2024-04-25T19:12:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc2bfff640 1 -- 172.21.15.28:0/2042453754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc24032440 con 0x7fbc340b5130 2024-04-25T19:12:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc2bfff640 1 -- 172.21.15.28:0/2042453754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24032810 con 0x7fbc340b5130 2024-04-25T19:12:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc3ba34640 1 -- 172.21.15.28:0/2042453754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc340b5130 msgr2=0x7fbc34161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/2042453754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc340b5130 0x7fbc34161130 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbc240097a0 tx=0x7fbc2402a180 comp rx=0 tx=0).stop 2024-04-25T19:12:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc3ba34640 1 -- 172.21.15.28:0/2042453754 shutdown_connections 2024-04-25T19:12:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/2042453754 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc3406ded0 0x7fbc3406e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/2042453754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc340b5130 0x7fbc34161130 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.521+0000 7fbc3ba34640 1 -- 172.21.15.28:0/2042453754 >> 172.21.15.28:0/2042453754 conn(0x7fbc340b7ad0 msgr2=0x7fbc340b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.522+0000 7fbc3ba34640 1 -- 172.21.15.28:0/2042453754 shutdown_connections 2024-04-25T19:12:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.522+0000 7fbc3ba34640 1 -- 172.21.15.28:0/2042453754 wait complete. 2024-04-25T19:12:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.523+0000 7fbc3ba34640 1 Processor -- start 2024-04-25T19:12:48.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.523+0000 7fbc3ba34640 1 -- start start 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc3ba34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc3406ded0 0x7fbc34156e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc3ba34640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc340b5130 0x7fbc3415d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc3ba34640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc340d0830 0x7fbc3415dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc3ba34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc34164720 con 0x7fbc3406ded0 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc3ba34640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbc34164440 con 0x7fbc340b5130 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc3ba34640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbc341607e0 con 0x7fbc340d0830 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc3406ded0 0x7fbc34156e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc3406ded0 0x7fbc34156e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54100/0 (socket says 172.21.15.28:54100) 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 -- 172.21.15.28:0/1273047998 learned_addr learned my addr 172.21.15.28:0/1273047998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc39fd1640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc340d0830 0x7fbc3415dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc38fcf640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc340b5130 0x7fbc3415d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 -- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc340d0830 msgr2=0x7fbc3415dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc340d0830 0x7fbc3415dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 -- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc340b5130 msgr2=0x7fbc3415d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc340b5130 0x7fbc3415d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 -- 172.21.15.28:0/1273047998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc341f6e00 con 0x7fbc3406ded0 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc38fcf640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc340b5130 0x7fbc3415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc397d0640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc3406ded0 0x7fbc34156e80 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbc240098c0 tx=0x7fbc240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc29ffb640 1 -- 172.21.15.28:0/1273047998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24041030 con 0x7fbc3406ded0 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc29ffb640 1 -- 172.21.15.28:0/1273047998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc2403d4e0 con 0x7fbc3406ded0 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc29ffb640 1 -- 172.21.15.28:0/1273047998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc24037040 con 0x7fbc3406ded0 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.525+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc341f70e0 con 0x7fbc3406ded0 2024-04-25T19:12:48.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.526+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc341f79f0 con 0x7fbc3406ded0 2024-04-25T19:12:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.527+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc34050740 con 0x7fbc3406ded0 2024-04-25T19:12:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.527+0000 7fbc29ffb640 1 -- 172.21.15.28:0/1273047998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbc24044070 con 0x7fbc3406ded0 2024-04-25T19:12:48.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.528+0000 7fbc29ffb640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc180744b0 0x7fbc180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.528+0000 7fbc29ffb640 1 -- 172.21.15.28:0/1273047998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fbc240b5a40 con 0x7fbc3406ded0 2024-04-25T19:12:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.528+0000 7fbc38fcf640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc180744b0 0x7fbc180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.528+0000 7fbc38fcf640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc180744b0 0x7fbc180769c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbc340689a0 tx=0x7fbc30008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.532+0000 7fbc29ffb640 1 -- 172.21.15.28:0/1273047998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc24080040 con 0x7fbc3406ded0 2024-04-25T19:12:48.653 INFO:tasks.ceph.mds.b.smithi053.stderr:2024-04-25T19:12:48.652+0000 7efe21111640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T19:12:48.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.713+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc340d2fe0 con 0x7fbc180744b0 2024-04-25T19:12:48.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.719+0000 7fbc29ffb640 1 -- 172.21.15.28:0/1273047998 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251907 (secure 0 0 0) 0x7fbc340d2fe0 con 0x7fbc180744b0 2024-04-25T19:12:48.725 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc180744b0 msgr2=0x7fbc180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc180744b0 0x7fbc180769c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fbc340689a0 tx=0x7fbc30008040 comp rx=0 tx=0).stop 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc3406ded0 msgr2=0x7fbc34156e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc3406ded0 0x7fbc34156e80 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbc240098c0 tx=0x7fbc240046c0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 shutdown_connections 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc180744b0 0x7fbc180769c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc340d0830 0x7fbc3415dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc340b5130 0x7fbc3415d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 --2- 172.21.15.28:0/1273047998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc3406ded0 0x7fbc34156e80 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.722+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 >> 172.21.15.28:0/1273047998 conn(0x7fbc340b7ad0 msgr2=0x7fbc340d45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.723+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 shutdown_connections 2024-04-25T19:12:48.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.723+0000 7fbc3ba34640 1 -- 172.21.15.28:0/1273047998 wait complete. 2024-04-25T19:12:48.735 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-04-25T19:12:46.853004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254000,"kb_used_data":4552,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720720,"statfs":{"total":773094113280,"available":772834017280,"internally_reserved":0,"allocated":4661248,"data_stored":2652244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.190744"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.704911+0000","last_change":"2024-04-25T19:12:44.700653+0000","last_active":"2024-04-25T19:12:44.704911+0000","last_peered":"2024-04-25T19:12:44.704911+0000","last_clean":"2024-04-25T19:12:44.704911+0000","last_became_active":"2024-04-25T19:12:44.700326+0000","last_became_peered":"2024-04-25T19:12:44.700326+0000","last_unstale":"2024-04-25T19:12:44.704911+0000","last_undegraded":"2024-04-25T19:12:44.704911+0000","last_fullsized":"2024-04-25T19:12:44.704911+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:29.311080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702147+0000","last_change":"2024-04-25T19:12:44.699053+0000","last_active":"2024-04-25T19:12:44.702147+0000","last_peered":"2024-04-25T19:12:44.702147+0000","last_clean":"2024-04-25T19:12:44.702147+0000","last_became_active":"2024-04-25T19:12:44.698718+0000","last_became_peered":"2024-04-25T19:12:44.698718+0000","last_unstale":"2024-04-25T19:12:44.702147+0000","last_undegraded":"2024-04-25T19:12:44.702147+0000","last_fullsized":"2024-04-25T19:12:44.702147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:58.680446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.695580+0000","last_change":"2024-04-25T19:12:44.692595+0000","last_active":"2024-04-25T19:12:44.695580+0000","last_peered":"2024-04-25T19:12:44.695580+0000","last_clean":"2024-04-25T19:12:44.695580+0000","last_became_active":"2024-04-25T19:12:44.692288+0000","last_became_peered":"2024-04-25T19:12:44.692288+0000","last_unstale":"2024-04-25T19:12:44.695580+0000","last_undegraded":"2024-04-25T19:12:44.695580+0000","last_fullsized":"2024-04-25T19:12:44.695580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:45.225366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.697128+0000","last_change":"2024-04-25T19:12:44.691657+0000","last_active":"2024-04-25T19:12:44.697128+0000","last_peered":"2024-04-25T19:12:44.697128+0000","last_clean":"2024-04-25T19:12:44.697128+0000","last_became_active":"2024-04-25T19:12:44.691153+0000","last_became_peered":"2024-04-25T19:12:44.691153+0000","last_unstale":"2024-04-25T19:12:44.697128+0000","last_undegraded":"2024-04-25T19:12:44.697128+0000","last_fullsized":"2024-04-25T19:12:44.697128+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:06.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.689547+0000","last_change":"2024-04-25T19:12:44.685404+0000","last_active":"2024-04-25T19:12:44.689547+0000","last_peered":"2024-04-25T19:12:44.689547+0000","last_clean":"2024-04-25T19:12:44.689547+0000","last_became_active":"2024-04-25T19:12:44.685079+0000","last_became_peered":"2024-04-25T19:12:44.685079+0000","last_unstale":"2024-04-25T19:12:44.689547+0000","last_undegraded":"2024-04-25T19:12:44.689547+0000","last_fullsized":"2024-04-25T19:12:44.689547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:34:35.956098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.703488+0000","last_change":"2024-04-25T19:12:44.702769+0000","last_active":"2024-04-25T19:12:44.703488+0000","last_peered":"2024-04-25T19:12:44.703488+0000","last_clean":"2024-04-25T19:12:44.703488+0000","last_became_active":"2024-04-25T19:12:44.702437+0000","last_became_peered":"2024-04-25T19:12:44.702437+0000","last_unstale":"2024-04-25T19:12:44.703488+0000","last_undegraded":"2024-04-25T19:12:44.703488+0000","last_fullsized":"2024-04-25T19:12:44.703488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:17.038220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690293+0000","last_change":"2024-04-25T19:12:44.705815+0000","last_active":"2024-04-25T19:12:45.690293+0000","last_peered":"2024-04-25T19:12:45.690293+0000","last_clean":"2024-04-25T19:12:45.690293+0000","last_became_active":"2024-04-25T19:12:44.705457+0000","last_became_peered":"2024-04-25T19:12:44.705457+0000","last_unstale":"2024-04-25T19:12:45.690293+0000","last_undegraded":"2024-04-25T19:12:45.690293+0000","last_fullsized":"2024-04-25T19:12:45.690293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.704217+0000","last_change":"2024-04-25T19:12:44.701020+0000","last_active":"2024-04-25T19:12:44.704217+0000","last_peered":"2024-04-25T19:12:44.704217+0000","last_clean":"2024-04-25T19:12:44.704217+0000","last_became_active":"2024-04-25T19:12:44.700672+0000","last_became_peered":"2024-04-25T19:12:44.700672+0000","last_unstale":"2024-04-25T19:12:44.704217+0000","last_undegraded":"2024-04-25T19:12:44.704217+0000","last_fullsized":"2024-04-25T19:12:44.704217+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:57.582503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.696282+0000","last_change":"2024-04-25T19:12:44.693607+0000","last_active":"2024-04-25T19:12:44.696282+0000","last_peered":"2024-04-25T19:12:44.696282+0000","last_clean":"2024-04-25T19:12:44.696282+0000","last_became_active":"2024-04-25T19:12:44.693303+0000","last_became_peered":"2024-04-25T19:12:44.693303+0000","last_unstale":"2024-04-25T19:12:44.696282+0000","last_undegraded":"2024-04-25T19:12:44.696282+0000","last_fullsized":"2024-04-25T19:12:44.696282+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:22.202361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.693534+0000","last_change":"2024-04-25T19:12:44.695584+0000","last_active":"2024-04-25T19:12:45.693534+0000","last_peered":"2024-04-25T19:12:45.693534+0000","last_clean":"2024-04-25T19:12:45.693534+0000","last_became_active":"2024-04-25T19:12:44.695174+0000","last_became_peered":"2024-04-25T19:12:44.695174+0000","last_unstale":"2024-04-25T19:12:45.693534+0000","last_undegraded":"2024-04-25T19:12:45.693534+0000","last_fullsized":"2024-04-25T19:12:45.693534+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:36.123098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692196+0000","last_change":"2024-04-25T19:12:44.692511+0000","last_active":"2024-04-25T19:12:45.692196+0000","last_peered":"2024-04-25T19:12:45.692196+0000","last_clean":"2024-04-25T19:12:45.692196+0000","last_became_active":"2024-04-25T19:12:44.692175+0000","last_became_peered":"2024-04-25T19:12:44.692175+0000","last_unstale":"2024-04-25T19:12:45.692196+0000","last_undegraded":"2024-04-25T19:12:45.692196+0000","last_fullsized":"2024-04-25T19:12:45.692196+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:55:25.129409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.697861+0000","last_change":"2024-04-25T19:12:44.691012+0000","last_active":"2024-04-25T19:12:44.697861+0000","last_peered":"2024-04-25T19:12:44.697861+0000","last_clean":"2024-04-25T19:12:44.697861+0000","last_became_active":"2024-04-25T19:12:44.690627+0000","last_became_peered":"2024-04-25T19:12:44.690627+0000","last_unstale":"2024-04-25T19:12:44.697861+0000","last_undegraded":"2024-04-25T19:12:44.697861+0000","last_fullsized":"2024-04-25T19:12:44.697861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:00.405188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691313+0000","last_change":"2024-04-25T19:12:44.696238+0000","last_active":"2024-04-25T19:12:45.691313+0000","last_peered":"2024-04-25T19:12:45.691313+0000","last_clean":"2024-04-25T19:12:45.691313+0000","last_became_active":"2024-04-25T19:12:44.695884+0000","last_became_peered":"2024-04-25T19:12:44.695884+0000","last_unstale":"2024-04-25T19:12:45.691313+0000","last_undegraded":"2024-04-25T19:12:45.691313+0000","last_fullsized":"2024-04-25T19:12:45.691313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:16.882127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.697821+0000","last_change":"2024-04-25T19:12:44.690998+0000","last_active":"2024-04-25T19:12:44.697821+0000","last_peered":"2024-04-25T19:12:44.697821+0000","last_clean":"2024-04-25T19:12:44.697821+0000","last_became_active":"2024-04-25T19:12:44.690126+0000","last_became_peered":"2024-04-25T19:12:44.690126+0000","last_unstale":"2024-04-25T19:12:44.697821+0000","last_undegraded":"2024-04-25T19:12:44.697821+0000","last_fullsized":"2024-04-25T19:12:44.697821+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:56.264599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690495+0000","last_change":"2024-04-25T19:12:44.681433+0000","last_active":"2024-04-25T19:12:45.690495+0000","last_peered":"2024-04-25T19:12:45.690495+0000","last_clean":"2024-04-25T19:12:45.690495+0000","last_became_active":"2024-04-25T19:12:44.681100+0000","last_became_peered":"2024-04-25T19:12:44.681100+0000","last_unstale":"2024-04-25T19:12:45.690495+0000","last_undegraded":"2024-04-25T19:12:45.690495+0000","last_fullsized":"2024-04-25T19:12:45.690495+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:22.449658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692811+0000","last_change":"2024-04-25T19:12:44.701150+0000","last_active":"2024-04-25T19:12:45.692811+0000","last_peered":"2024-04-25T19:12:45.692811+0000","last_clean":"2024-04-25T19:12:45.692811+0000","last_became_active":"2024-04-25T19:12:44.699787+0000","last_became_peered":"2024-04-25T19:12:44.699787+0000","last_unstale":"2024-04-25T19:12:45.692811+0000","last_undegraded":"2024-04-25T19:12:45.692811+0000","last_fullsized":"2024-04-25T19:12:45.692811+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:12.662654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692930+0000","last_change":"2024-04-25T19:12:44.706630+0000","last_active":"2024-04-25T19:12:45.692930+0000","last_peered":"2024-04-25T19:12:45.692930+0000","last_clean":"2024-04-25T19:12:45.692930+0000","last_became_active":"2024-04-25T19:12:44.705055+0000","last_became_peered":"2024-04-25T19:12:44.705055+0000","last_unstale":"2024-04-25T19:12:45.692930+0000","last_undegraded":"2024-04-25T19:12:45.692930+0000","last_fullsized":"2024-04-25T19:12:45.692930+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690859+0000","last_change":"2024-04-25T19:12:44.683787+0000","last_active":"2024-04-25T19:12:45.690859+0000","last_peered":"2024-04-25T19:12:45.690859+0000","last_clean":"2024-04-25T19:12:45.690859+0000","last_became_active":"2024-04-25T19:12:44.683469+0000","last_became_peered":"2024-04-25T19:12:44.683469+0000","last_unstale":"2024-04-25T19:12:45.690859+0000","last_undegraded":"2024-04-25T19:12:45.690859+0000","last_fullsized":"2024-04-25T19:12:45.690859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:34.463674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.684670+0000","last_change":"2024-04-25T19:12:44.683518+0000","last_active":"2024-04-25T19:12:44.684670+0000","last_peered":"2024-04-25T19:12:44.684670+0000","last_clean":"2024-04-25T19:12:44.684670+0000","last_became_active":"2024-04-25T19:12:44.683194+0000","last_became_peered":"2024-04-25T19:12:44.683194+0000","last_unstale":"2024-04-25T19:12:44.684670+0000","last_undegraded":"2024-04-25T19:12:44.684670+0000","last_fullsized":"2024-04-25T19:12:44.684670+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:53:50.270010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690980+0000","last_change":"2024-04-25T19:12:44.697957+0000","last_active":"2024-04-25T19:12:45.690980+0000","last_peered":"2024-04-25T19:12:45.690980+0000","last_clean":"2024-04-25T19:12:45.690980+0000","last_became_active":"2024-04-25T19:12:44.697633+0000","last_became_peered":"2024-04-25T19:12:44.697633+0000","last_unstale":"2024-04-25T19:12:45.690980+0000","last_undegraded":"2024-04-25T19:12:45.690980+0000","last_fullsized":"2024-04-25T19:12:45.690980+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:24.118674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691501+0000","last_change":"2024-04-25T19:12:44.690591+0000","last_active":"2024-04-25T19:12:45.691501+0000","last_peered":"2024-04-25T19:12:45.691501+0000","last_clean":"2024-04-25T19:12:45.691501+0000","last_became_active":"2024-04-25T19:12:44.690265+0000","last_became_peered":"2024-04-25T19:12:44.690265+0000","last_unstale":"2024-04-25T19:12:45.691501+0000","last_undegraded":"2024-04-25T19:12:45.691501+0000","last_fullsized":"2024-04-25T19:12:45.691501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:40.972621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.689845+0000","last_change":"2024-04-25T19:12:44.683217+0000","last_active":"2024-04-25T19:12:44.689845+0000","last_peered":"2024-04-25T19:12:44.689845+0000","last_clean":"2024-04-25T19:12:44.689845+0000","last_became_active":"2024-04-25T19:12:44.682860+0000","last_became_peered":"2024-04-25T19:12:44.682860+0000","last_unstale":"2024-04-25T19:12:44.689845+0000","last_undegraded":"2024-04-25T19:12:44.689845+0000","last_fullsized":"2024-04-25T19:12:44.689845+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:48.266667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691646+0000","last_change":"2024-04-25T19:12:44.704757+0000","last_active":"2024-04-25T19:12:45.691646+0000","last_peered":"2024-04-25T19:12:45.691646+0000","last_clean":"2024-04-25T19:12:45.691646+0000","last_became_active":"2024-04-25T19:12:44.704345+0000","last_became_peered":"2024-04-25T19:12:44.704345+0000","last_unstale":"2024-04-25T19:12:45.691646+0000","last_undegraded":"2024-04-25T19:12:45.691646+0000","last_fullsized":"2024-04-25T19:12:45.691646+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:05.288431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.689513+0000","last_change":"2024-04-25T19:12:44.685369+0000","last_active":"2024-04-25T19:12:44.689513+0000","last_peered":"2024-04-25T19:12:44.689513+0000","last_clean":"2024-04-25T19:12:44.689513+0000","last_became_active":"2024-04-25T19:12:44.684749+0000","last_became_peered":"2024-04-25T19:12:44.684749+0000","last_unstale":"2024-04-25T19:12:44.689513+0000","last_undegraded":"2024-04-25T19:12:44.689513+0000","last_fullsized":"2024-04-25T19:12:44.689513+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:44.938696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691201+0000","last_change":"2024-04-25T19:12:44.688826+0000","last_active":"2024-04-25T19:12:45.691201+0000","last_peered":"2024-04-25T19:12:45.691201+0000","last_clean":"2024-04-25T19:12:45.691201+0000","last_became_active":"2024-04-25T19:12:44.687859+0000","last_became_peered":"2024-04-25T19:12:44.687859+0000","last_unstale":"2024-04-25T19:12:45.691201+0000","last_undegraded":"2024-04-25T19:12:45.691201+0000","last_fullsized":"2024-04-25T19:12:45.691201+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:00.405188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702084+0000","last_change":"2024-04-25T19:12:44.698551+0000","last_active":"2024-04-25T19:12:44.702084+0000","last_peered":"2024-04-25T19:12:44.702084+0000","last_clean":"2024-04-25T19:12:44.702084+0000","last_became_active":"2024-04-25T19:12:44.698111+0000","last_became_peered":"2024-04-25T19:12:44.698111+0000","last_unstale":"2024-04-25T19:12:44.702084+0000","last_undegraded":"2024-04-25T19:12:44.702084+0000","last_fullsized":"2024-04-25T19:12:44.702084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:45.225366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691523+0000","last_change":"2024-04-25T19:12:44.681953+0000","last_active":"2024-04-25T19:12:45.691523+0000","last_peered":"2024-04-25T19:12:45.691523+0000","last_clean":"2024-04-25T19:12:45.691523+0000","last_became_active":"2024-04-25T19:12:44.681238+0000","last_became_peered":"2024-04-25T19:12:44.681238+0000","last_unstale":"2024-04-25T19:12:45.691523+0000","last_undegraded":"2024-04-25T19:12:45.691523+0000","last_fullsized":"2024-04-25T19:12:45.691523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:37.425520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692024+0000","last_change":"2024-04-25T19:12:44.706575+0000","last_active":"2024-04-25T19:12:45.692024+0000","last_peered":"2024-04-25T19:12:45.692024+0000","last_clean":"2024-04-25T19:12:45.692024+0000","last_became_active":"2024-04-25T19:12:44.706268+0000","last_became_peered":"2024-04-25T19:12:44.706268+0000","last_unstale":"2024-04-25T19:12:45.692024+0000","last_undegraded":"2024-04-25T19:12:45.692024+0000","last_fullsized":"2024-04-25T19:12:45.692024+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:21.849594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691487+0000","last_change":"2024-04-25T19:12:44.682967+0000","last_active":"2024-04-25T19:12:45.691487+0000","last_peered":"2024-04-25T19:12:45.691487+0000","last_clean":"2024-04-25T19:12:45.691487+0000","last_became_active":"2024-04-25T19:12:44.682477+0000","last_became_peered":"2024-04-25T19:12:44.682477+0000","last_unstale":"2024-04-25T19:12:45.691487+0000","last_undegraded":"2024-04-25T19:12:45.691487+0000","last_fullsized":"2024-04-25T19:12:45.691487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:45.225366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692065+0000","last_change":"2024-04-25T19:12:44.699586+0000","last_active":"2024-04-25T19:12:45.692065+0000","last_peered":"2024-04-25T19:12:45.692065+0000","last_clean":"2024-04-25T19:12:45.692065+0000","last_became_active":"2024-04-25T19:12:44.699289+0000","last_became_peered":"2024-04-25T19:12:44.699289+0000","last_unstale":"2024-04-25T19:12:45.692065+0000","last_undegraded":"2024-04-25T19:12:45.692065+0000","last_fullsized":"2024-04-25T19:12:45.692065+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:22.202361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691741+0000","last_change":"2024-04-25T19:12:44.702160+0000","last_active":"2024-04-25T19:12:45.691741+0000","last_peered":"2024-04-25T19:12:45.691741+0000","last_clean":"2024-04-25T19:12:45.691741+0000","last_became_active":"2024-04-25T19:12:44.701815+0000","last_became_peered":"2024-04-25T19:12:44.701815+0000","last_unstale":"2024-04-25T19:12:45.691741+0000","last_undegraded":"2024-04-25T19:12:45.691741+0000","last_fullsized":"2024-04-25T19:12:45.691741+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:45.995608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.692617+0000","last_change":"2024-04-25T19:12:44.691655+0000","last_active":"2024-04-25T19:12:44.692617+0000","last_peered":"2024-04-25T19:12:44.692617+0000","last_clean":"2024-04-25T19:12:44.692617+0000","last_became_active":"2024-04-25T19:12:44.691354+0000","last_became_peered":"2024-04-25T19:12:44.691354+0000","last_unstale":"2024-04-25T19:12:44.692617+0000","last_undegraded":"2024-04-25T19:12:44.692617+0000","last_fullsized":"2024-04-25T19:12:44.692617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:00.405188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691421+0000","last_change":"2024-04-25T19:12:44.705515+0000","last_active":"2024-04-25T19:12:45.691421+0000","last_peered":"2024-04-25T19:12:45.691421+0000","last_clean":"2024-04-25T19:12:45.691421+0000","last_became_active":"2024-04-25T19:12:44.703843+0000","last_became_peered":"2024-04-25T19:12:44.703843+0000","last_unstale":"2024-04-25T19:12:45.691421+0000","last_undegraded":"2024-04-25T19:12:45.691421+0000","last_fullsized":"2024-04-25T19:12:45.691421+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:28.034410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691329+0000","last_change":"2024-04-25T19:12:43.687421+0000","last_active":"2024-04-25T19:12:45.691329+0000","last_peered":"2024-04-25T19:12:45.691329+0000","last_clean":"2024-04-25T19:12:45.691329+0000","last_became_active":"2024-04-25T19:12:43.687150+0000","last_became_peered":"2024-04-25T19:12:43.687150+0000","last_unstale":"2024-04-25T19:12:45.691329+0000","last_undegraded":"2024-04-25T19:12:45.691329+0000","last_fullsized":"2024-04-25T19:12:45.691329+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:22.014467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702763+0000","last_change":"2024-04-25T19:12:44.698702+0000","last_active":"2024-04-25T19:12:44.702763+0000","last_peered":"2024-04-25T19:12:44.702763+0000","last_clean":"2024-04-25T19:12:44.702763+0000","last_became_active":"2024-04-25T19:12:44.698378+0000","last_became_peered":"2024-04-25T19:12:44.698378+0000","last_unstale":"2024-04-25T19:12:44.702763+0000","last_undegraded":"2024-04-25T19:12:44.702763+0000","last_fullsized":"2024-04-25T19:12:44.702763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:37.425520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.708263+0000","last_change":"2024-04-25T19:12:43.687002+0000","last_active":"2024-04-25T19:12:44.708263+0000","last_peered":"2024-04-25T19:12:44.708263+0000","last_clean":"2024-04-25T19:12:44.708263+0000","last_became_active":"2024-04-25T19:12:43.686681+0000","last_became_peered":"2024-04-25T19:12:43.686681+0000","last_unstale":"2024-04-25T19:12:44.708263+0000","last_undegraded":"2024-04-25T19:12:44.708263+0000","last_fullsized":"2024-04-25T19:12:44.708263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.690406+0000","last_change":"2024-04-25T19:12:44.688116+0000","last_active":"2024-04-25T19:12:44.690406+0000","last_peered":"2024-04-25T19:12:44.690406+0000","last_clean":"2024-04-25T19:12:44.690406+0000","last_became_active":"2024-04-25T19:12:44.687776+0000","last_became_peered":"2024-04-25T19:12:44.687776+0000","last_unstale":"2024-04-25T19:12:44.690406+0000","last_undegraded":"2024-04-25T19:12:44.690406+0000","last_fullsized":"2024-04-25T19:12:44.690406+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:32:51.909308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.701918+0000","last_change":"2024-04-25T19:12:43.682264+0000","last_active":"2024-04-25T19:12:44.701918+0000","last_peered":"2024-04-25T19:12:44.701918+0000","last_clean":"2024-04-25T19:12:44.701918+0000","last_became_active":"2024-04-25T19:12:43.681918+0000","last_became_peered":"2024-04-25T19:12:43.681918+0000","last_unstale":"2024-04-25T19:12:44.701918+0000","last_undegraded":"2024-04-25T19:12:44.701918+0000","last_fullsized":"2024-04-25T19:12:44.701918+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:55.257918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702148+0000","last_change":"2024-04-25T19:12:44.699758+0000","last_active":"2024-04-25T19:12:44.702148+0000","last_peered":"2024-04-25T19:12:44.702148+0000","last_clean":"2024-04-25T19:12:44.702148+0000","last_became_active":"2024-04-25T19:12:44.699294+0000","last_became_peered":"2024-04-25T19:12:44.699294+0000","last_unstale":"2024-04-25T19:12:44.702148+0000","last_undegraded":"2024-04-25T19:12:44.702148+0000","last_fullsized":"2024-04-25T19:12:44.702148+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:45.995608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.708446+0000","last_change":"2024-04-25T19:12:43.690606+0000","last_active":"2024-04-25T19:12:44.708446+0000","last_peered":"2024-04-25T19:12:44.708446+0000","last_clean":"2024-04-25T19:12:44.708446+0000","last_became_active":"2024-04-25T19:12:43.690163+0000","last_became_peered":"2024-04-25T19:12:43.690163+0000","last_unstale":"2024-04-25T19:12:44.708446+0000","last_undegraded":"2024-04-25T19:12:44.708446+0000","last_fullsized":"2024-04-25T19:12:44.708446+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:44.282277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.696212+0000","last_change":"2024-04-25T19:12:44.692946+0000","last_active":"2024-04-25T19:12:44.696212+0000","last_peered":"2024-04-25T19:12:44.696212+0000","last_clean":"2024-04-25T19:12:44.696212+0000","last_became_active":"2024-04-25T19:12:44.692070+0000","last_became_peered":"2024-04-25T19:12:44.692070+0000","last_unstale":"2024-04-25T19:12:44.696212+0000","last_undegraded":"2024-04-25T19:12:44.696212+0000","last_fullsized":"2024-04-25T19:12:44.696212+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:37.425520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691647+0000","last_change":"2024-04-25T19:12:43.682355+0000","last_active":"2024-04-25T19:12:45.691647+0000","last_peered":"2024-04-25T19:12:45.691647+0000","last_clean":"2024-04-25T19:12:45.691647+0000","last_became_active":"2024-04-25T19:12:43.682041+0000","last_became_peered":"2024-04-25T19:12:43.682041+0000","last_unstale":"2024-04-25T19:12:45.691647+0000","last_undegraded":"2024-04-25T19:12:45.691647+0000","last_fullsized":"2024-04-25T19:12:45.691647+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:44.218685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690685+0000","last_change":"2024-04-25T19:12:44.691213+0000","last_active":"2024-04-25T19:12:45.690685+0000","last_peered":"2024-04-25T19:12:45.690685+0000","last_clean":"2024-04-25T19:12:45.690685+0000","last_became_active":"2024-04-25T19:12:44.690834+0000","last_became_peered":"2024-04-25T19:12:44.690834+0000","last_unstale":"2024-04-25T19:12:45.690685+0000","last_undegraded":"2024-04-25T19:12:45.690685+0000","last_fullsized":"2024-04-25T19:12:45.690685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:17.267296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690316+0000","last_change":"2024-04-25T19:12:43.677641+0000","last_active":"2024-04-25T19:12:45.690316+0000","last_peered":"2024-04-25T19:12:45.690316+0000","last_clean":"2024-04-25T19:12:45.690316+0000","last_became_active":"2024-04-25T19:12:43.677313+0000","last_became_peered":"2024-04-25T19:12:43.677313+0000","last_unstale":"2024-04-25T19:12:45.690316+0000","last_undegraded":"2024-04-25T19:12:45.690316+0000","last_fullsized":"2024-04-25T19:12:45.690316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:26.370845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.700978+0000","last_change":"2024-04-25T19:12:44.694711+0000","last_active":"2024-04-25T19:12:44.700978+0000","last_peered":"2024-04-25T19:12:44.700978+0000","last_clean":"2024-04-25T19:12:44.700978+0000","last_became_active":"2024-04-25T19:12:44.693493+0000","last_became_peered":"2024-04-25T19:12:44.693493+0000","last_unstale":"2024-04-25T19:12:44.700978+0000","last_undegraded":"2024-04-25T19:12:44.700978+0000","last_fullsized":"2024-04-25T19:12:44.700978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:54.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.678419+0000","last_change":"2024-04-25T19:12:43.691781+0000","last_active":"2024-04-25T19:12:44.678419+0000","last_peered":"2024-04-25T19:12:44.678419+0000","last_clean":"2024-04-25T19:12:44.678419+0000","last_became_active":"2024-04-25T19:12:43.691468+0000","last_became_peered":"2024-04-25T19:12:43.691468+0000","last_unstale":"2024-04-25T19:12:44.678419+0000","last_undegraded":"2024-04-25T19:12:44.678419+0000","last_fullsized":"2024-04-25T19:12:44.678419+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:26.370845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.706217+0000","last_change":"2024-04-25T19:12:44.703258+0000","last_active":"2024-04-25T19:12:44.706217+0000","last_peered":"2024-04-25T19:12:44.706217+0000","last_clean":"2024-04-25T19:12:44.706217+0000","last_became_active":"2024-04-25T19:12:44.702920+0000","last_became_peered":"2024-04-25T19:12:44.702920+0000","last_unstale":"2024-04-25T19:12:44.706217+0000","last_undegraded":"2024-04-25T19:12:44.706217+0000","last_fullsized":"2024-04-25T19:12:44.706217+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.684150+0000","last_change":"2024-04-25T19:12:43.687531+0000","last_active":"2024-04-25T19:12:44.684150+0000","last_peered":"2024-04-25T19:12:44.684150+0000","last_clean":"2024-04-25T19:12:44.684150+0000","last_became_active":"2024-04-25T19:12:43.687230+0000","last_became_peered":"2024-04-25T19:12:43.687230+0000","last_unstale":"2024-04-25T19:12:44.684150+0000","last_undegraded":"2024-04-25T19:12:44.684150+0000","last_fullsized":"2024-04-25T19:12:44.684150+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:22.014467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.685624+0000","last_change":"2024-04-25T19:12:44.684353+0000","last_active":"2024-04-25T19:12:44.685624+0000","last_peered":"2024-04-25T19:12:44.685624+0000","last_clean":"2024-04-25T19:12:44.685624+0000","last_became_active":"2024-04-25T19:12:44.684024+0000","last_became_peered":"2024-04-25T19:12:44.684024+0000","last_unstale":"2024-04-25T19:12:44.685624+0000","last_undegraded":"2024-04-25T19:12:44.685624+0000","last_fullsized":"2024-04-25T19:12:44.685624+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:35.384793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.686423+0000","last_change":"2024-04-25T19:12:43.685597+0000","last_active":"2024-04-25T19:12:44.686423+0000","last_peered":"2024-04-25T19:12:44.686423+0000","last_clean":"2024-04-25T19:12:44.686423+0000","last_became_active":"2024-04-25T19:12:43.685245+0000","last_became_peered":"2024-04-25T19:12:43.685245+0000","last_unstale":"2024-04-25T19:12:44.686423+0000","last_undegraded":"2024-04-25T19:12:44.686423+0000","last_fullsized":"2024-04-25T19:12:44.686423+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:35.845516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692014+0000","last_change":"2024-04-25T19:12:44.698686+0000","last_active":"2024-04-25T19:12:45.692014+0000","last_peered":"2024-04-25T19:12:45.692014+0000","last_clean":"2024-04-25T19:12:45.692014+0000","last_became_active":"2024-04-25T19:12:44.697877+0000","last_became_peered":"2024-04-25T19:12:44.697877+0000","last_unstale":"2024-04-25T19:12:45.692014+0000","last_undegraded":"2024-04-25T19:12:45.692014+0000","last_fullsized":"2024-04-25T19:12:45.692014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:29.311080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.693689+0000","last_change":"2024-04-25T19:12:43.691155+0000","last_active":"2024-04-25T19:12:45.693689+0000","last_peered":"2024-04-25T19:12:45.693689+0000","last_clean":"2024-04-25T19:12:45.693689+0000","last_became_active":"2024-04-25T19:12:43.690735+0000","last_became_peered":"2024-04-25T19:12:43.690735+0000","last_unstale":"2024-04-25T19:12:45.693689+0000","last_undegraded":"2024-04-25T19:12:45.693689+0000","last_fullsized":"2024-04-25T19:12:45.693689+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:36.418839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.698998+0000","last_change":"2024-04-25T19:12:44.698058+0000","last_active":"2024-04-25T19:12:44.698998+0000","last_peered":"2024-04-25T19:12:44.698998+0000","last_clean":"2024-04-25T19:12:44.698998+0000","last_became_active":"2024-04-25T19:12:44.695384+0000","last_became_peered":"2024-04-25T19:12:44.695384+0000","last_unstale":"2024-04-25T19:12:44.698998+0000","last_undegraded":"2024-04-25T19:12:44.698998+0000","last_fullsized":"2024-04-25T19:12:44.698998+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:08:46.997852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692326+0000","last_change":"2024-04-25T19:12:43.686664+0000","last_active":"2024-04-25T19:12:45.692326+0000","last_peered":"2024-04-25T19:12:45.692326+0000","last_clean":"2024-04-25T19:12:45.692326+0000","last_became_active":"2024-04-25T19:12:43.686375+0000","last_became_peered":"2024-04-25T19:12:43.686375+0000","last_unstale":"2024-04-25T19:12:45.692326+0000","last_undegraded":"2024-04-25T19:12:45.692326+0000","last_fullsized":"2024-04-25T19:12:45.692326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:35.845516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.691231+0000","last_change":"2024-04-25T19:12:44.690212+0000","last_active":"2024-04-25T19:12:44.691231+0000","last_peered":"2024-04-25T19:12:44.691231+0000","last_clean":"2024-04-25T19:12:44.691231+0000","last_became_active":"2024-04-25T19:12:44.689709+0000","last_became_peered":"2024-04-25T19:12:44.689709+0000","last_unstale":"2024-04-25T19:12:44.691231+0000","last_undegraded":"2024-04-25T19:12:44.691231+0000","last_fullsized":"2024-04-25T19:12:44.691231+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:27.377526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.710832+0000","last_change":"2024-04-25T19:12:43.689503+0000","last_active":"2024-04-25T19:12:44.710832+0000","last_peered":"2024-04-25T19:12:44.710832+0000","last_clean":"2024-04-25T19:12:44.710832+0000","last_became_active":"2024-04-25T19:12:43.689203+0000","last_became_peered":"2024-04-25T19:12:43.689203+0000","last_unstale":"2024-04-25T19:12:44.710832+0000","last_undegraded":"2024-04-25T19:12:44.710832+0000","last_fullsized":"2024-04-25T19:12:44.710832+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:08:45.991172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.696291+0000","last_change":"2024-04-25T19:12:44.693888+0000","last_active":"2024-04-25T19:12:44.696291+0000","last_peered":"2024-04-25T19:12:44.696291+0000","last_clean":"2024-04-25T19:12:44.696291+0000","last_became_active":"2024-04-25T19:12:44.691696+0000","last_became_peered":"2024-04-25T19:12:44.691696+0000","last_unstale":"2024-04-25T19:12:44.696291+0000","last_undegraded":"2024-04-25T19:12:44.696291+0000","last_fullsized":"2024-04-25T19:12:44.696291+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:17.038220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.701994+0000","last_change":"2024-04-25T19:12:43.691713+0000","last_active":"2024-04-25T19:12:44.701994+0000","last_peered":"2024-04-25T19:12:44.701994+0000","last_clean":"2024-04-25T19:12:44.701994+0000","last_became_active":"2024-04-25T19:12:43.691375+0000","last_became_peered":"2024-04-25T19:12:43.691375+0000","last_unstale":"2024-04-25T19:12:44.701994+0000","last_undegraded":"2024-04-25T19:12:44.701994+0000","last_fullsized":"2024-04-25T19:12:44.701994+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:58.179259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702082+0000","last_change":"2024-04-25T19:12:44.699213+0000","last_active":"2024-04-25T19:12:44.702082+0000","last_peered":"2024-04-25T19:12:44.702082+0000","last_clean":"2024-04-25T19:12:44.702082+0000","last_became_active":"2024-04-25T19:12:44.698893+0000","last_became_peered":"2024-04-25T19:12:44.698893+0000","last_unstale":"2024-04-25T19:12:44.702082+0000","last_undegraded":"2024-04-25T19:12:44.702082+0000","last_fullsized":"2024-04-25T19:12:44.702082+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:54.272820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.678812+0000","last_change":"2024-04-25T19:12:43.691127+0000","last_active":"2024-04-25T19:12:44.678812+0000","last_peered":"2024-04-25T19:12:44.678812+0000","last_clean":"2024-04-25T19:12:44.678812+0000","last_became_active":"2024-04-25T19:12:43.690372+0000","last_became_peered":"2024-04-25T19:12:43.690372+0000","last_unstale":"2024-04-25T19:12:44.678812+0000","last_undegraded":"2024-04-25T19:12:44.678812+0000","last_fullsized":"2024-04-25T19:12:44.678812+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690328+0000","last_change":"2024-04-25T19:12:44.698557+0000","last_active":"2024-04-25T19:12:45.690328+0000","last_peered":"2024-04-25T19:12:45.690328+0000","last_clean":"2024-04-25T19:12:45.690328+0000","last_became_active":"2024-04-25T19:12:44.698170+0000","last_became_peered":"2024-04-25T19:12:44.698170+0000","last_unstale":"2024-04-25T19:12:45.690328+0000","last_undegraded":"2024-04-25T19:12:45.690328+0000","last_fullsized":"2024-04-25T19:12:45.690328+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:58.680446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692316+0000","last_change":"2024-04-25T19:12:43.687570+0000","last_active":"2024-04-25T19:12:45.692316+0000","last_peered":"2024-04-25T19:12:45.692316+0000","last_clean":"2024-04-25T19:12:45.692316+0000","last_became_active":"2024-04-25T19:12:43.687216+0000","last_became_peered":"2024-04-25T19:12:43.687216+0000","last_unstale":"2024-04-25T19:12:45.692316+0000","last_undegraded":"2024-04-25T19:12:45.692316+0000","last_fullsized":"2024-04-25T19:12:45.692316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:56.285928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702120+0000","last_change":"2024-04-25T19:12:44.699754+0000","last_active":"2024-04-25T19:12:44.702120+0000","last_peered":"2024-04-25T19:12:44.702120+0000","last_clean":"2024-04-25T19:12:44.702120+0000","last_became_active":"2024-04-25T19:12:44.699422+0000","last_became_peered":"2024-04-25T19:12:44.699422+0000","last_unstale":"2024-04-25T19:12:44.702120+0000","last_undegraded":"2024-04-25T19:12:44.702120+0000","last_fullsized":"2024-04-25T19:12:44.702120+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:05.288431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692389+0000","last_change":"2024-04-25T19:12:43.677420+0000","last_active":"2024-04-25T19:12:45.692389+0000","last_peered":"2024-04-25T19:12:45.692389+0000","last_clean":"2024-04-25T19:12:45.692389+0000","last_became_active":"2024-04-25T19:12:43.677108+0000","last_became_peered":"2024-04-25T19:12:43.677108+0000","last_unstale":"2024-04-25T19:12:45.692389+0000","last_undegraded":"2024-04-25T19:12:45.692389+0000","last_fullsized":"2024-04-25T19:12:45.692389+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:32:50.902627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702208+0000","last_change":"2024-04-25T19:12:44.699717+0000","last_active":"2024-04-25T19:12:44.702208+0000","last_peered":"2024-04-25T19:12:44.702208+0000","last_clean":"2024-04-25T19:12:44.702208+0000","last_became_active":"2024-04-25T19:12:44.699361+0000","last_became_peered":"2024-04-25T19:12:44.699361+0000","last_unstale":"2024-04-25T19:12:44.702208+0000","last_undegraded":"2024-04-25T19:12:44.702208+0000","last_fullsized":"2024-04-25T19:12:44.702208+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:28.034410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692338+0000","last_change":"2024-04-25T19:12:43.679044+0000","last_active":"2024-04-25T19:12:45.692338+0000","last_peered":"2024-04-25T19:12:45.692338+0000","last_clean":"2024-04-25T19:12:45.692338+0000","last_became_active":"2024-04-25T19:12:43.678723+0000","last_became_peered":"2024-04-25T19:12:43.678723+0000","last_unstale":"2024-04-25T19:12:45.692338+0000","last_undegraded":"2024-04-25T19:12:45.692338+0000","last_fullsized":"2024-04-25T19:12:45.692338+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:16.031539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692310+0000","last_change":"2024-04-25T19:12:44.697124+0000","last_active":"2024-04-25T19:12:45.692310+0000","last_peered":"2024-04-25T19:12:45.692310+0000","last_clean":"2024-04-25T19:12:45.692310+0000","last_became_active":"2024-04-25T19:12:44.695351+0000","last_became_peered":"2024-04-25T19:12:44.695351+0000","last_unstale":"2024-04-25T19:12:45.692310+0000","last_undegraded":"2024-04-25T19:12:45.692310+0000","last_fullsized":"2024-04-25T19:12:45.692310+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:24.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.697383+0000","last_change":"2024-04-25T19:12:43.678051+0000","last_active":"2024-04-25T19:12:44.697383+0000","last_peered":"2024-04-25T19:12:44.697383+0000","last_clean":"2024-04-25T19:12:44.697383+0000","last_became_active":"2024-04-25T19:12:43.677640+0000","last_became_peered":"2024-04-25T19:12:43.677640+0000","last_unstale":"2024-04-25T19:12:44.697383+0000","last_undegraded":"2024-04-25T19:12:44.697383+0000","last_fullsized":"2024-04-25T19:12:44.697383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:06.360629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.700904+0000","last_change":"2024-04-25T19:12:44.695748+0000","last_active":"2024-04-25T19:12:44.700904+0000","last_peered":"2024-04-25T19:12:44.700904+0000","last_clean":"2024-04-25T19:12:44.700904+0000","last_became_active":"2024-04-25T19:12:44.695447+0000","last_became_peered":"2024-04-25T19:12:44.695447+0000","last_unstale":"2024-04-25T19:12:44.700904+0000","last_undegraded":"2024-04-25T19:12:44.700904+0000","last_fullsized":"2024-04-25T19:12:44.700904+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:34.463674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.708696+0000","last_change":"2024-04-25T19:12:43.684026+0000","last_active":"2024-04-25T19:12:44.708696+0000","last_peered":"2024-04-25T19:12:44.708696+0000","last_clean":"2024-04-25T19:12:44.708696+0000","last_became_active":"2024-04-25T19:12:43.683693+0000","last_became_peered":"2024-04-25T19:12:43.683693+0000","last_unstale":"2024-04-25T19:12:44.708696+0000","last_undegraded":"2024-04-25T19:12:44.708696+0000","last_fullsized":"2024-04-25T19:12:44.708696+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:14.924183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.703516+0000","last_change":"2024-04-25T19:12:44.701567+0000","last_active":"2024-04-25T19:12:44.703516+0000","last_peered":"2024-04-25T19:12:44.703516+0000","last_clean":"2024-04-25T19:12:44.703516+0000","last_became_active":"2024-04-25T19:12:44.701201+0000","last_became_peered":"2024-04-25T19:12:44.701201+0000","last_unstale":"2024-04-25T19:12:44.703516+0000","last_undegraded":"2024-04-25T19:12:44.703516+0000","last_fullsized":"2024-04-25T19:12:44.703516+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:27.013579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.679214+0000","last_change":"2024-04-25T19:12:43.684026+0000","last_active":"2024-04-25T19:12:44.679214+0000","last_peered":"2024-04-25T19:12:44.679214+0000","last_clean":"2024-04-25T19:12:44.679214+0000","last_became_active":"2024-04-25T19:12:43.683700+0000","last_became_peered":"2024-04-25T19:12:43.683700+0000","last_unstale":"2024-04-25T19:12:44.679214+0000","last_undegraded":"2024-04-25T19:12:44.679214+0000","last_fullsized":"2024-04-25T19:12:44.679214+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:33.456993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692569+0000","last_change":"2024-04-25T19:12:44.699373+0000","last_active":"2024-04-25T19:12:45.692569+0000","last_peered":"2024-04-25T19:12:45.692569+0000","last_clean":"2024-04-25T19:12:45.692569+0000","last_became_active":"2024-04-25T19:12:44.693763+0000","last_became_peered":"2024-04-25T19:12:44.693763+0000","last_unstale":"2024-04-25T19:12:45.692569+0000","last_undegraded":"2024-04-25T19:12:45.692569+0000","last_fullsized":"2024-04-25T19:12:45.692569+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.701058+0000","last_change":"2024-04-25T19:12:43.687714+0000","last_active":"2024-04-25T19:12:44.701058+0000","last_peered":"2024-04-25T19:12:44.701058+0000","last_clean":"2024-04-25T19:12:44.701058+0000","last_became_active":"2024-04-25T19:12:43.687375+0000","last_became_peered":"2024-04-25T19:12:43.687375+0000","last_unstale":"2024-04-25T19:12:44.701058+0000","last_undegraded":"2024-04-25T19:12:44.701058+0000","last_fullsized":"2024-04-25T19:12:44.701058+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:01.568678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.696878+0000","last_change":"2024-04-25T19:12:44.694953+0000","last_active":"2024-04-25T19:12:44.696878+0000","last_peered":"2024-04-25T19:12:44.696878+0000","last_clean":"2024-04-25T19:12:44.696878+0000","last_became_active":"2024-04-25T19:12:44.694607+0000","last_became_peered":"2024-04-25T19:12:44.694607+0000","last_unstale":"2024-04-25T19:12:44.696878+0000","last_undegraded":"2024-04-25T19:12:44.696878+0000","last_fullsized":"2024-04-25T19:12:44.696878+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:36.852197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.671698+0000","last_change":"2024-04-25T19:12:43.688266+0000","last_active":"2024-04-25T19:12:44.671698+0000","last_peered":"2024-04-25T19:12:44.671698+0000","last_clean":"2024-04-25T19:12:44.671698+0000","last_became_active":"2024-04-25T19:12:43.687912+0000","last_became_peered":"2024-04-25T19:12:43.687912+0000","last_unstale":"2024-04-25T19:12:44.671698+0000","last_undegraded":"2024-04-25T19:12:44.671698+0000","last_fullsized":"2024-04-25T19:12:44.671698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:54.337918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691090+0000","last_change":"2024-04-25T19:12:44.687324+0000","last_active":"2024-04-25T19:12:45.691090+0000","last_peered":"2024-04-25T19:12:45.691090+0000","last_clean":"2024-04-25T19:12:45.691090+0000","last_became_active":"2024-04-25T19:12:44.686274+0000","last_became_peered":"2024-04-25T19:12:44.686274+0000","last_unstale":"2024-04-25T19:12:45.691090+0000","last_undegraded":"2024-04-25T19:12:45.691090+0000","last_fullsized":"2024-04-25T19:12:45.691090+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:20.222641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691774+0000","last_change":"2024-04-25T19:12:43.687265+0000","last_active":"2024-04-25T19:12:45.691774+0000","last_peered":"2024-04-25T19:12:45.691774+0000","last_clean":"2024-04-25T19:12:45.691774+0000","last_became_active":"2024-04-25T19:12:43.686948+0000","last_became_peered":"2024-04-25T19:12:43.686948+0000","last_unstale":"2024-04-25T19:12:45.691774+0000","last_undegraded":"2024-04-25T19:12:45.691774+0000","last_fullsized":"2024-04-25T19:12:45.691774+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:55.257918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.706746+0000","last_change":"2024-04-25T19:12:44.706030+0000","last_active":"2024-04-25T19:12:44.706746+0000","last_peered":"2024-04-25T19:12:44.706746+0000","last_clean":"2024-04-25T19:12:44.706746+0000","last_became_active":"2024-04-25T19:12:44.704826+0000","last_became_peered":"2024-04-25T19:12:44.704826+0000","last_unstale":"2024-04-25T19:12:44.706746+0000","last_undegraded":"2024-04-25T19:12:44.706746+0000","last_fullsized":"2024-04-25T19:12:44.706746+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.671506+0000","last_change":"2024-04-25T19:12:43.688066+0000","last_active":"2024-04-25T19:12:44.671506+0000","last_peered":"2024-04-25T19:12:44.671506+0000","last_clean":"2024-04-25T19:12:44.671506+0000","last_became_active":"2024-04-25T19:12:43.687694+0000","last_became_peered":"2024-04-25T19:12:43.687694+0000","last_unstale":"2024-04-25T19:12:44.671506+0000","last_undegraded":"2024-04-25T19:12:44.671506+0000","last_fullsized":"2024-04-25T19:12:44.671506+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:24.803044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702122+0000","last_change":"2024-04-25T19:12:44.698861+0000","last_active":"2024-04-25T19:12:44.702122+0000","last_peered":"2024-04-25T19:12:44.702122+0000","last_clean":"2024-04-25T19:12:44.702122+0000","last_became_active":"2024-04-25T19:12:44.698178+0000","last_became_peered":"2024-04-25T19:12:44.698178+0000","last_unstale":"2024-04-25T19:12:44.702122+0000","last_undegraded":"2024-04-25T19:12:44.702122+0000","last_fullsized":"2024-04-25T19:12:44.702122+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:22.202361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692565+0000","last_change":"2024-04-25T19:12:43.685799+0000","last_active":"2024-04-25T19:12:45.692565+0000","last_peered":"2024-04-25T19:12:45.692565+0000","last_clean":"2024-04-25T19:12:45.692565+0000","last_became_active":"2024-04-25T19:12:43.685478+0000","last_became_peered":"2024-04-25T19:12:43.685478+0000","last_unstale":"2024-04-25T19:12:45.692565+0000","last_undegraded":"2024-04-25T19:12:45.692565+0000","last_fullsized":"2024-04-25T19:12:45.692565+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:21.195681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692175+0000","last_change":"2024-04-25T19:12:44.699033+0000","last_active":"2024-04-25T19:12:45.692175+0000","last_peered":"2024-04-25T19:12:45.692175+0000","last_clean":"2024-04-25T19:12:45.692175+0000","last_became_active":"2024-04-25T19:12:44.698376+0000","last_became_peered":"2024-04-25T19:12:44.698376+0000","last_unstale":"2024-04-25T19:12:45.692175+0000","last_undegraded":"2024-04-25T19:12:45.692175+0000","last_fullsized":"2024-04-25T19:12:45.692175+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:57.582503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.672020+0000","last_change":"2024-04-25T19:12:43.688186+0000","last_active":"2024-04-25T19:12:44.672020+0000","last_peered":"2024-04-25T19:12:44.672020+0000","last_clean":"2024-04-25T19:12:44.672020+0000","last_became_active":"2024-04-25T19:12:43.687361+0000","last_became_peered":"2024-04-25T19:12:43.687361+0000","last_unstale":"2024-04-25T19:12:44.672020+0000","last_undegraded":"2024-04-25T19:12:44.672020+0000","last_fullsized":"2024-04-25T19:12:44.672020+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:37.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.706462+0000","last_change":"2024-04-25T19:12:44.705586+0000","last_active":"2024-04-25T19:12:44.706462+0000","last_peered":"2024-04-25T19:12:44.706462+0000","last_clean":"2024-04-25T19:12:44.706462+0000","last_became_active":"2024-04-25T19:12:44.704667+0000","last_became_peered":"2024-04-25T19:12:44.704667+0000","last_unstale":"2024-04-25T19:12:44.706462+0000","last_undegraded":"2024-04-25T19:12:44.706462+0000","last_fullsized":"2024-04-25T19:12:44.706462+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:12.662654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.678072+0000","last_change":"2024-04-25T19:12:43.685345+0000","last_active":"2024-04-25T19:12:44.678072+0000","last_peered":"2024-04-25T19:12:44.678072+0000","last_clean":"2024-04-25T19:12:44.678072+0000","last_became_active":"2024-04-25T19:12:43.684968+0000","last_became_peered":"2024-04-25T19:12:43.684968+0000","last_unstale":"2024-04-25T19:12:44.678072+0000","last_undegraded":"2024-04-25T19:12:44.678072+0000","last_fullsized":"2024-04-25T19:12:44.678072+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:33.456993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.704151+0000","last_change":"2024-04-25T19:12:44.702919+0000","last_active":"2024-04-25T19:12:44.704151+0000","last_peered":"2024-04-25T19:12:44.704151+0000","last_clean":"2024-04-25T19:12:44.704151+0000","last_became_active":"2024-04-25T19:12:44.702600+0000","last_became_peered":"2024-04-25T19:12:44.702600+0000","last_unstale":"2024-04-25T19:12:44.704151+0000","last_undegraded":"2024-04-25T19:12:44.704151+0000","last_fullsized":"2024-04-25T19:12:44.704151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:59.185940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.678903+0000","last_change":"2024-04-25T19:12:43.689072+0000","last_active":"2024-04-25T19:12:44.678903+0000","last_peered":"2024-04-25T19:12:44.678903+0000","last_clean":"2024-04-25T19:12:44.678903+0000","last_became_active":"2024-04-25T19:12:43.688665+0000","last_became_peered":"2024-04-25T19:12:43.688665+0000","last_unstale":"2024-04-25T19:12:44.678903+0000","last_undegraded":"2024-04-25T19:12:44.678903+0000","last_fullsized":"2024-04-25T19:12:44.678903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:23.111993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.689342+0000","last_change":"2024-04-25T19:12:44.683442+0000","last_active":"2024-04-25T19:12:44.689342+0000","last_peered":"2024-04-25T19:12:44.689342+0000","last_clean":"2024-04-25T19:12:44.689342+0000","last_became_active":"2024-04-25T19:12:44.683137+0000","last_became_peered":"2024-04-25T19:12:44.683137+0000","last_unstale":"2024-04-25T19:12:44.689342+0000","last_undegraded":"2024-04-25T19:12:44.689342+0000","last_fullsized":"2024-04-25T19:12:44.689342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:24.118674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692192+0000","last_change":"2024-04-25T19:12:43.688566+0000","last_active":"2024-04-25T19:12:45.692192+0000","last_peered":"2024-04-25T19:12:45.692192+0000","last_clean":"2024-04-25T19:12:45.692192+0000","last_became_active":"2024-04-25T19:12:43.688009+0000","last_became_peered":"2024-04-25T19:12:43.688009+0000","last_unstale":"2024-04-25T19:12:45.692192+0000","last_undegraded":"2024-04-25T19:12:45.692192+0000","last_fullsized":"2024-04-25T19:12:45.692192+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:47.259986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.708536+0000","last_change":"2024-04-25T19:12:44.704959+0000","last_active":"2024-04-25T19:12:44.708536+0000","last_peered":"2024-04-25T19:12:44.708536+0000","last_clean":"2024-04-25T19:12:44.708536+0000","last_became_active":"2024-04-25T19:12:44.704200+0000","last_became_peered":"2024-04-25T19:12:44.704200+0000","last_unstale":"2024-04-25T19:12:44.708536+0000","last_undegraded":"2024-04-25T19:12:44.708536+0000","last_fullsized":"2024-04-25T19:12:44.708536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:24.118674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.684579+0000","last_change":"2024-04-25T19:12:43.686166+0000","last_active":"2024-04-25T19:12:44.684579+0000","last_peered":"2024-04-25T19:12:44.684579+0000","last_clean":"2024-04-25T19:12:44.684579+0000","last_became_active":"2024-04-25T19:12:43.685724+0000","last_became_peered":"2024-04-25T19:12:43.685724+0000","last_unstale":"2024-04-25T19:12:44.684579+0000","last_undegraded":"2024-04-25T19:12:44.684579+0000","last_fullsized":"2024-04-25T19:12:44.684579+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:00:37.087793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691109+0000","last_change":"2024-04-25T19:12:44.702272+0000","last_active":"2024-04-25T19:12:45.691109+0000","last_peered":"2024-04-25T19:12:45.691109+0000","last_clean":"2024-04-25T19:12:45.691109+0000","last_became_active":"2024-04-25T19:12:44.701778+0000","last_became_peered":"2024-04-25T19:12:44.701778+0000","last_unstale":"2024-04-25T19:12:45.691109+0000","last_undegraded":"2024-04-25T19:12:45.691109+0000","last_fullsized":"2024-04-25T19:12:45.691109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:54.272820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.693344+0000","last_change":"2024-04-25T19:12:43.686223+0000","last_active":"2024-04-25T19:12:44.693344+0000","last_peered":"2024-04-25T19:12:44.693344+0000","last_clean":"2024-04-25T19:12:44.693344+0000","last_became_active":"2024-04-25T19:12:43.685881+0000","last_became_peered":"2024-04-25T19:12:43.685881+0000","last_unstale":"2024-04-25T19:12:44.693344+0000","last_undegraded":"2024-04-25T19:12:44.693344+0000","last_fullsized":"2024-04-25T19:12:44.693344+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:08.114375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692651+0000","last_change":"2024-04-25T19:12:44.706605+0000","last_active":"2024-04-25T19:12:45.692651+0000","last_peered":"2024-04-25T19:12:45.692651+0000","last_clean":"2024-04-25T19:12:45.692651+0000","last_became_active":"2024-04-25T19:12:44.706192+0000","last_became_peered":"2024-04-25T19:12:44.706192+0000","last_unstale":"2024-04-25T19:12:45.692651+0000","last_undegraded":"2024-04-25T19:12:45.692651+0000","last_fullsized":"2024-04-25T19:12:45.692651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:34.452188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691542+0000","last_change":"2024-04-25T19:12:43.689029+0000","last_active":"2024-04-25T19:12:45.691542+0000","last_peered":"2024-04-25T19:12:45.691542+0000","last_clean":"2024-04-25T19:12:45.691542+0000","last_became_active":"2024-04-25T19:12:43.688728+0000","last_became_peered":"2024-04-25T19:12:43.688728+0000","last_unstale":"2024-04-25T19:12:45.691542+0000","last_undegraded":"2024-04-25T19:12:45.691542+0000","last_fullsized":"2024-04-25T19:12:45.691542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":141,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692639+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:12:45.692639+0000","last_peered":"2024-04-25T19:12:45.692639+0000","last_clean":"2024-04-25T19:12:45.692639+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:12:45.692639+0000","last_undegraded":"2024-04-25T19:12:45.692639+0000","last_fullsized":"2024-04-25T19:12:45.692639+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1036288,"data_stored":775386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31600,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":430080,"data_stored":183918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31604,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340236,"statfs":{"total":96636764160,"available":96604401664,"internally_reserved":0,"allocated":434176,"data_stored":183448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31604,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340236,"statfs":{"total":96636764160,"available":96604401664,"internally_reserved":0,"allocated":434176,"data_stored":183940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":184982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31600,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":430080,"data_stored":183406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31600,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":430080,"data_stored":183426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1024000,"data_stored":773738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:12:48.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:12:48.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.889+0000 7fbcb8042640 1 Processor -- start 2024-04-25T19:12:48.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.889+0000 7fbcb8042640 1 -- start start 2024-04-25T19:12:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.890+0000 7fbcb8042640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbcb00b7d70 0x7fbcb015dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.890+0000 7fbcb8042640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcb01606f0 0x7fbcb0160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.890+0000 7fbcb8042640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcb01614a0 con 0x7fbcb0164990 2024-04-25T19:12:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.890+0000 7fbcb8042640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fbcb0163eb0 con 0x7fbcb00b7d70 2024-04-25T19:12:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.890+0000 7fbcb8042640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb0164190 con 0x7fbcb01606f0 2024-04-25T19:12:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.890+0000 7fbcb5dde640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbcb0164990 0x7fbcb00b7660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39748/0 (socket says 172.21.15.28:39748) 2024-04-25T19:12:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.890+0000 7fbcb5dde640 1 -- 172.21.15.28:0/1396583890 learned_addr learned my addr 172.21.15.28:0/1396583890 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 937032996 0 0) 0x7fbcb01614a0 con 0x7fbcb0164990 2024-04-25T19:12:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb55dd640 1 --2- 172.21.15.28:0/1396583890 >> v2:172.21.15.53:3300/0 conn(0x7fbcb00b7d70 0x7fbcb015dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbc9c003520 con 0x7fbcb0164990 2024-04-25T19:12:48.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb65df640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcb01606f0 0x7fbcb0160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1853554383 0 0) 0x7fbc9c003520 con 0x7fbcb0164990 2024-04-25T19:12:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcb01606f0 msgr2=0x7fbcb0160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcb01606f0 0x7fbcb0160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 >> v2:172.21.15.53:3300/0 conn(0x7fbcb00b7d70 msgr2=0x7fbcb015dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 --2- 172.21.15.28:0/1396583890 >> v2:172.21.15.53:3300/0 conn(0x7fbcb00b7d70 0x7fbcb015dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb0164470 con 0x7fbcb0164990 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fbcac001690 con 0x7fbcb0164990 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 >> v1:172.21.15.28:6789/0 conn(0x7fbcb0164990 legacy=0x7fbcb00b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c0039b0 0x7fbc9c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9c004800 0x7fbc9c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcb01606f0 0x7fbc9c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0163eb0 con 0x7fbc9c004800 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0164190 con 0x7fbc9c0039b0 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9c014200 con 0x7fbcb01606f0 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb55dd640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9c004800 0x7fbc9c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb5dde640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c0039b0 0x7fbc9c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb65df640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcb01606f0 0x7fbc9c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb55dd640 1 -- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcb01606f0 msgr2=0x7fbc9c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb55dd640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcb01606f0 0x7fbc9c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb55dd640 1 -- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c0039b0 msgr2=0x7fbc9c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb55dd640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc9c0039b0 0x7fbc9c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.891+0000 7fbcb55dd640 1 -- 172.21.15.28:0/1396583890 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9c003520 con 0x7fbc9c004800 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.892+0000 7fbcb55dd640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9c004800 0x7fbc9c004c10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009ac0 tx=0x7fbc9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd2ccc29db76a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.892+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0036070 con 0x7fbc9c004800 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.892+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca0031400 con 0x7fbc9c004800 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.892+0000 7fbcb4ddc640 1 -- 172.21.15.28:0/1396583890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca0009690 con 0x7fbc9c004800 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.893+0000 7fbcb8042640 1 -- 172.21.15.28:0/1396583890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9c004800 msgr2=0x7fbc9c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.893+0000 7fbcb8042640 1 --2- 172.21.15.28:0/1396583890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9c004800 0x7fbc9c004c10 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fbca0009ac0 tx=0x7fbc9c005830 comp rx=0 tx=0).stop 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.893+0000 7fbcb65df640 1 -- 172.21.15.28:0/1396583890 reap_dead start 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.893+0000 7fbcb8042640 1 -- 172.21.15.28:0/1396583890 shutdown_connections 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.893+0000 7fbcb8042640 1 -- 172.21.15.28:0/1396583890 >> 172.21.15.28:0/1396583890 conn(0x7fbcb00d2840 msgr2=0x7fbcb00d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.893+0000 7fbcb8042640 1 -- 172.21.15.28:0/1396583890 shutdown_connections 2024-04-25T19:12:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.893+0000 7fbcb8042640 1 -- 172.21.15.28:0/1396583890 wait complete. 2024-04-25T19:12:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.895+0000 7fbcb8042640 1 Processor -- start 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.895+0000 7fbcb8042640 1 -- start start 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.896+0000 7fbcb8042640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c004800 0x7fbcb015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.896+0000 7fbcb8042640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb015d050 0x7fbcb00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.896+0000 7fbcb8042640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb00cf8b0 0x7fbcb00cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.896+0000 7fbcb8042640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0160ed0 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.896+0000 7fbcb8042640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0160d50 con 0x7fbcb015d050 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.896+0000 7fbcb8042640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb0161050 con 0x7fbc9c004800 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.896+0000 7fbcb65df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb00cf8b0 0x7fbcb00cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb65df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb00cf8b0 0x7fbcb00cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54120/0 (socket says 172.21.15.28:54120) 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb65df640 1 -- 172.21.15.28:0/1903971007 learned_addr learned my addr 172.21.15.28:0/1903971007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb5dde640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c004800 0x7fbcb015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb55dd640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb015d050 0x7fbcb00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb65df640 1 -- 172.21.15.28:0/1903971007 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c004800 msgr2=0x7fbcb015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb65df640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c004800 0x7fbcb015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb65df640 1 -- 172.21.15.28:0/1903971007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb015d050 msgr2=0x7fbcb00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb65df640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb015d050 0x7fbcb00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb65df640 1 -- 172.21.15.28:0/1903971007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb01622e0 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb65df640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb00cf8b0 0x7fbcb00cdd00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00c96e0 tx=0x7fbca80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbca67fc640 1 -- 172.21.15.28:0/1903971007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8003990 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbca67fc640 1 -- 172.21.15.28:0/1903971007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca801b070 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb00ccee0 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbca67fc640 1 -- 172.21.15.28:0/1903971007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8003b70 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.897+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb0162580 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.898+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcb0050740 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.899+0000 7fbca67fc640 1 -- 172.21.15.28:0/1903971007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbca8005d10 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.900+0000 7fbca67fc640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc940744b0 0x7fbc940769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.900+0000 7fbcb5dde640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc940744b0 0x7fbc940769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.900+0000 7fbca67fc640 1 -- 172.21.15.28:0/1903971007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fbca80956a0 con 0x7fbcb00cf8b0 2024-04-25T19:12:48.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.900+0000 7fbcb5dde640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc940744b0 0x7fbc940769c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00d5bb0 tx=0x7fbcac007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:48.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:48.905+0000 7fbca67fc640 1 -- 172.21.15.28:0/1903971007 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbca805f650 con 0x7fbcb00cf8b0 2024-04-25T19:12:49.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.086+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbcb00ccc90 con 0x7fbc940744b0 2024-04-25T19:12:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.092+0000 7fbca67fc640 1 -- 172.21.15.28:0/1903971007 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251907 (secure 0 0 0) 0x7fbcb00ccc90 con 0x7fbc940744b0 2024-04-25T19:12:49.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:12:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc940744b0 msgr2=0x7fbc940769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc940744b0 0x7fbc940769c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00d5bb0 tx=0x7fbcac007040 comp rx=0 tx=0).stop 2024-04-25T19:12:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb00cf8b0 msgr2=0x7fbcb00cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb00cf8b0 0x7fbcb00cdd00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fbcb00c96e0 tx=0x7fbca80115a0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 shutdown_connections 2024-04-25T19:12:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbc940744b0 0x7fbc940769c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcb00cf8b0 0x7fbcb00cdd00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbcb015d050 0x7fbcb00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 --2- 172.21.15.28:0/1903971007 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbc9c004800 0x7fbcb015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 >> 172.21.15.28:0/1903971007 conn(0x7fbcb00d2840 msgr2=0x7fbcb00d5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 shutdown_connections 2024-04-25T19:12:49.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.095+0000 7fbcb8042640 1 -- 172.21.15.28:0/1903971007 wait complete. 2024-04-25T19:12:49.108 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-25T19:12:48.853506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":129,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254000,"kb_used_data":4552,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720720,"statfs":{"total":773094113280,"available":772834017280,"internally_reserved":0,"allocated":4661248,"data_stored":2652244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001151"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.704911+0000","last_change":"2024-04-25T19:12:44.700653+0000","last_active":"2024-04-25T19:12:44.704911+0000","last_peered":"2024-04-25T19:12:44.704911+0000","last_clean":"2024-04-25T19:12:44.704911+0000","last_became_active":"2024-04-25T19:12:44.700326+0000","last_became_peered":"2024-04-25T19:12:44.700326+0000","last_unstale":"2024-04-25T19:12:44.704911+0000","last_undegraded":"2024-04-25T19:12:44.704911+0000","last_fullsized":"2024-04-25T19:12:44.704911+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:29.311080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702147+0000","last_change":"2024-04-25T19:12:44.699053+0000","last_active":"2024-04-25T19:12:44.702147+0000","last_peered":"2024-04-25T19:12:44.702147+0000","last_clean":"2024-04-25T19:12:44.702147+0000","last_became_active":"2024-04-25T19:12:44.698718+0000","last_became_peered":"2024-04-25T19:12:44.698718+0000","last_unstale":"2024-04-25T19:12:44.702147+0000","last_undegraded":"2024-04-25T19:12:44.702147+0000","last_fullsized":"2024-04-25T19:12:44.702147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:58.680446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.695580+0000","last_change":"2024-04-25T19:12:44.692595+0000","last_active":"2024-04-25T19:12:44.695580+0000","last_peered":"2024-04-25T19:12:44.695580+0000","last_clean":"2024-04-25T19:12:44.695580+0000","last_became_active":"2024-04-25T19:12:44.692288+0000","last_became_peered":"2024-04-25T19:12:44.692288+0000","last_unstale":"2024-04-25T19:12:44.695580+0000","last_undegraded":"2024-04-25T19:12:44.695580+0000","last_fullsized":"2024-04-25T19:12:44.695580+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:45.225366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.697128+0000","last_change":"2024-04-25T19:12:44.691657+0000","last_active":"2024-04-25T19:12:44.697128+0000","last_peered":"2024-04-25T19:12:44.697128+0000","last_clean":"2024-04-25T19:12:44.697128+0000","last_became_active":"2024-04-25T19:12:44.691153+0000","last_became_peered":"2024-04-25T19:12:44.691153+0000","last_unstale":"2024-04-25T19:12:44.697128+0000","last_undegraded":"2024-04-25T19:12:44.697128+0000","last_fullsized":"2024-04-25T19:12:44.697128+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:11:06.485472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.689547+0000","last_change":"2024-04-25T19:12:44.685404+0000","last_active":"2024-04-25T19:12:44.689547+0000","last_peered":"2024-04-25T19:12:44.689547+0000","last_clean":"2024-04-25T19:12:44.689547+0000","last_became_active":"2024-04-25T19:12:44.685079+0000","last_became_peered":"2024-04-25T19:12:44.685079+0000","last_unstale":"2024-04-25T19:12:44.689547+0000","last_undegraded":"2024-04-25T19:12:44.689547+0000","last_fullsized":"2024-04-25T19:12:44.689547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:34:35.956098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.703488+0000","last_change":"2024-04-25T19:12:44.702769+0000","last_active":"2024-04-25T19:12:44.703488+0000","last_peered":"2024-04-25T19:12:44.703488+0000","last_clean":"2024-04-25T19:12:44.703488+0000","last_became_active":"2024-04-25T19:12:44.702437+0000","last_became_peered":"2024-04-25T19:12:44.702437+0000","last_unstale":"2024-04-25T19:12:44.703488+0000","last_undegraded":"2024-04-25T19:12:44.703488+0000","last_fullsized":"2024-04-25T19:12:44.703488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:17.038220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690293+0000","last_change":"2024-04-25T19:12:44.705815+0000","last_active":"2024-04-25T19:12:45.690293+0000","last_peered":"2024-04-25T19:12:45.690293+0000","last_clean":"2024-04-25T19:12:45.690293+0000","last_became_active":"2024-04-25T19:12:44.705457+0000","last_became_peered":"2024-04-25T19:12:44.705457+0000","last_unstale":"2024-04-25T19:12:45.690293+0000","last_undegraded":"2024-04-25T19:12:45.690293+0000","last_fullsized":"2024-04-25T19:12:45.690293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.704217+0000","last_change":"2024-04-25T19:12:44.701020+0000","last_active":"2024-04-25T19:12:44.704217+0000","last_peered":"2024-04-25T19:12:44.704217+0000","last_clean":"2024-04-25T19:12:44.704217+0000","last_became_active":"2024-04-25T19:12:44.700672+0000","last_became_peered":"2024-04-25T19:12:44.700672+0000","last_unstale":"2024-04-25T19:12:44.704217+0000","last_undegraded":"2024-04-25T19:12:44.704217+0000","last_fullsized":"2024-04-25T19:12:44.704217+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:57.582503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.696282+0000","last_change":"2024-04-25T19:12:44.693607+0000","last_active":"2024-04-25T19:12:44.696282+0000","last_peered":"2024-04-25T19:12:44.696282+0000","last_clean":"2024-04-25T19:12:44.696282+0000","last_became_active":"2024-04-25T19:12:44.693303+0000","last_became_peered":"2024-04-25T19:12:44.693303+0000","last_unstale":"2024-04-25T19:12:44.696282+0000","last_undegraded":"2024-04-25T19:12:44.696282+0000","last_fullsized":"2024-04-25T19:12:44.696282+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:22.202361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.693534+0000","last_change":"2024-04-25T19:12:44.695584+0000","last_active":"2024-04-25T19:12:45.693534+0000","last_peered":"2024-04-25T19:12:45.693534+0000","last_clean":"2024-04-25T19:12:45.693534+0000","last_became_active":"2024-04-25T19:12:44.695174+0000","last_became_peered":"2024-04-25T19:12:44.695174+0000","last_unstale":"2024-04-25T19:12:45.693534+0000","last_undegraded":"2024-04-25T19:12:45.693534+0000","last_fullsized":"2024-04-25T19:12:45.693534+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:36.123098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692196+0000","last_change":"2024-04-25T19:12:44.692511+0000","last_active":"2024-04-25T19:12:45.692196+0000","last_peered":"2024-04-25T19:12:45.692196+0000","last_clean":"2024-04-25T19:12:45.692196+0000","last_became_active":"2024-04-25T19:12:44.692175+0000","last_became_peered":"2024-04-25T19:12:44.692175+0000","last_unstale":"2024-04-25T19:12:45.692196+0000","last_undegraded":"2024-04-25T19:12:45.692196+0000","last_fullsized":"2024-04-25T19:12:45.692196+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:55:25.129409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.697861+0000","last_change":"2024-04-25T19:12:44.691012+0000","last_active":"2024-04-25T19:12:44.697861+0000","last_peered":"2024-04-25T19:12:44.697861+0000","last_clean":"2024-04-25T19:12:44.697861+0000","last_became_active":"2024-04-25T19:12:44.690627+0000","last_became_peered":"2024-04-25T19:12:44.690627+0000","last_unstale":"2024-04-25T19:12:44.697861+0000","last_undegraded":"2024-04-25T19:12:44.697861+0000","last_fullsized":"2024-04-25T19:12:44.697861+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:00.405188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691313+0000","last_change":"2024-04-25T19:12:44.696238+0000","last_active":"2024-04-25T19:12:45.691313+0000","last_peered":"2024-04-25T19:12:45.691313+0000","last_clean":"2024-04-25T19:12:45.691313+0000","last_became_active":"2024-04-25T19:12:44.695884+0000","last_became_peered":"2024-04-25T19:12:44.695884+0000","last_unstale":"2024-04-25T19:12:45.691313+0000","last_undegraded":"2024-04-25T19:12:45.691313+0000","last_fullsized":"2024-04-25T19:12:45.691313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:16.882127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.697821+0000","last_change":"2024-04-25T19:12:44.690998+0000","last_active":"2024-04-25T19:12:44.697821+0000","last_peered":"2024-04-25T19:12:44.697821+0000","last_clean":"2024-04-25T19:12:44.697821+0000","last_became_active":"2024-04-25T19:12:44.690126+0000","last_became_peered":"2024-04-25T19:12:44.690126+0000","last_unstale":"2024-04-25T19:12:44.697821+0000","last_undegraded":"2024-04-25T19:12:44.697821+0000","last_fullsized":"2024-04-25T19:12:44.697821+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:56.264599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690495+0000","last_change":"2024-04-25T19:12:44.681433+0000","last_active":"2024-04-25T19:12:45.690495+0000","last_peered":"2024-04-25T19:12:45.690495+0000","last_clean":"2024-04-25T19:12:45.690495+0000","last_became_active":"2024-04-25T19:12:44.681100+0000","last_became_peered":"2024-04-25T19:12:44.681100+0000","last_unstale":"2024-04-25T19:12:45.690495+0000","last_undegraded":"2024-04-25T19:12:45.690495+0000","last_fullsized":"2024-04-25T19:12:45.690495+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:22.449658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692811+0000","last_change":"2024-04-25T19:12:44.701150+0000","last_active":"2024-04-25T19:12:45.692811+0000","last_peered":"2024-04-25T19:12:45.692811+0000","last_clean":"2024-04-25T19:12:45.692811+0000","last_became_active":"2024-04-25T19:12:44.699787+0000","last_became_peered":"2024-04-25T19:12:44.699787+0000","last_unstale":"2024-04-25T19:12:45.692811+0000","last_undegraded":"2024-04-25T19:12:45.692811+0000","last_fullsized":"2024-04-25T19:12:45.692811+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:12.662654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692930+0000","last_change":"2024-04-25T19:12:44.706630+0000","last_active":"2024-04-25T19:12:45.692930+0000","last_peered":"2024-04-25T19:12:45.692930+0000","last_clean":"2024-04-25T19:12:45.692930+0000","last_became_active":"2024-04-25T19:12:44.705055+0000","last_became_peered":"2024-04-25T19:12:44.705055+0000","last_unstale":"2024-04-25T19:12:45.692930+0000","last_undegraded":"2024-04-25T19:12:45.692930+0000","last_fullsized":"2024-04-25T19:12:45.692930+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690859+0000","last_change":"2024-04-25T19:12:44.683787+0000","last_active":"2024-04-25T19:12:45.690859+0000","last_peered":"2024-04-25T19:12:45.690859+0000","last_clean":"2024-04-25T19:12:45.690859+0000","last_became_active":"2024-04-25T19:12:44.683469+0000","last_became_peered":"2024-04-25T19:12:44.683469+0000","last_unstale":"2024-04-25T19:12:45.690859+0000","last_undegraded":"2024-04-25T19:12:45.690859+0000","last_fullsized":"2024-04-25T19:12:45.690859+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:34.463674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.684670+0000","last_change":"2024-04-25T19:12:44.683518+0000","last_active":"2024-04-25T19:12:44.684670+0000","last_peered":"2024-04-25T19:12:44.684670+0000","last_clean":"2024-04-25T19:12:44.684670+0000","last_became_active":"2024-04-25T19:12:44.683194+0000","last_became_peered":"2024-04-25T19:12:44.683194+0000","last_unstale":"2024-04-25T19:12:44.684670+0000","last_undegraded":"2024-04-25T19:12:44.684670+0000","last_fullsized":"2024-04-25T19:12:44.684670+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:53:50.270010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690980+0000","last_change":"2024-04-25T19:12:44.697957+0000","last_active":"2024-04-25T19:12:45.690980+0000","last_peered":"2024-04-25T19:12:45.690980+0000","last_clean":"2024-04-25T19:12:45.690980+0000","last_became_active":"2024-04-25T19:12:44.697633+0000","last_became_peered":"2024-04-25T19:12:44.697633+0000","last_unstale":"2024-04-25T19:12:45.690980+0000","last_undegraded":"2024-04-25T19:12:45.690980+0000","last_fullsized":"2024-04-25T19:12:45.690980+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:24.118674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691501+0000","last_change":"2024-04-25T19:12:44.690591+0000","last_active":"2024-04-25T19:12:45.691501+0000","last_peered":"2024-04-25T19:12:45.691501+0000","last_clean":"2024-04-25T19:12:45.691501+0000","last_became_active":"2024-04-25T19:12:44.690265+0000","last_became_peered":"2024-04-25T19:12:44.690265+0000","last_unstale":"2024-04-25T19:12:45.691501+0000","last_undegraded":"2024-04-25T19:12:45.691501+0000","last_fullsized":"2024-04-25T19:12:45.691501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:13:40.972621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.689845+0000","last_change":"2024-04-25T19:12:44.683217+0000","last_active":"2024-04-25T19:12:44.689845+0000","last_peered":"2024-04-25T19:12:44.689845+0000","last_clean":"2024-04-25T19:12:44.689845+0000","last_became_active":"2024-04-25T19:12:44.682860+0000","last_became_peered":"2024-04-25T19:12:44.682860+0000","last_unstale":"2024-04-25T19:12:44.689845+0000","last_undegraded":"2024-04-25T19:12:44.689845+0000","last_fullsized":"2024-04-25T19:12:44.689845+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:48.266667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691646+0000","last_change":"2024-04-25T19:12:44.704757+0000","last_active":"2024-04-25T19:12:45.691646+0000","last_peered":"2024-04-25T19:12:45.691646+0000","last_clean":"2024-04-25T19:12:45.691646+0000","last_became_active":"2024-04-25T19:12:44.704345+0000","last_became_peered":"2024-04-25T19:12:44.704345+0000","last_unstale":"2024-04-25T19:12:45.691646+0000","last_undegraded":"2024-04-25T19:12:45.691646+0000","last_fullsized":"2024-04-25T19:12:45.691646+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:05.288431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.689513+0000","last_change":"2024-04-25T19:12:44.685369+0000","last_active":"2024-04-25T19:12:44.689513+0000","last_peered":"2024-04-25T19:12:44.689513+0000","last_clean":"2024-04-25T19:12:44.689513+0000","last_became_active":"2024-04-25T19:12:44.684749+0000","last_became_peered":"2024-04-25T19:12:44.684749+0000","last_unstale":"2024-04-25T19:12:44.689513+0000","last_undegraded":"2024-04-25T19:12:44.689513+0000","last_fullsized":"2024-04-25T19:12:44.689513+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:15:44.938696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691201+0000","last_change":"2024-04-25T19:12:44.688826+0000","last_active":"2024-04-25T19:12:45.691201+0000","last_peered":"2024-04-25T19:12:45.691201+0000","last_clean":"2024-04-25T19:12:45.691201+0000","last_became_active":"2024-04-25T19:12:44.687859+0000","last_became_peered":"2024-04-25T19:12:44.687859+0000","last_unstale":"2024-04-25T19:12:45.691201+0000","last_undegraded":"2024-04-25T19:12:45.691201+0000","last_fullsized":"2024-04-25T19:12:45.691201+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:00.405188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702084+0000","last_change":"2024-04-25T19:12:44.698551+0000","last_active":"2024-04-25T19:12:44.702084+0000","last_peered":"2024-04-25T19:12:44.702084+0000","last_clean":"2024-04-25T19:12:44.702084+0000","last_became_active":"2024-04-25T19:12:44.698111+0000","last_became_peered":"2024-04-25T19:12:44.698111+0000","last_unstale":"2024-04-25T19:12:44.702084+0000","last_undegraded":"2024-04-25T19:12:44.702084+0000","last_fullsized":"2024-04-25T19:12:44.702084+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:45.225366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691523+0000","last_change":"2024-04-25T19:12:44.681953+0000","last_active":"2024-04-25T19:12:45.691523+0000","last_peered":"2024-04-25T19:12:45.691523+0000","last_clean":"2024-04-25T19:12:45.691523+0000","last_became_active":"2024-04-25T19:12:44.681238+0000","last_became_peered":"2024-04-25T19:12:44.681238+0000","last_unstale":"2024-04-25T19:12:45.691523+0000","last_undegraded":"2024-04-25T19:12:45.691523+0000","last_fullsized":"2024-04-25T19:12:45.691523+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:37.425520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692024+0000","last_change":"2024-04-25T19:12:44.706575+0000","last_active":"2024-04-25T19:12:45.692024+0000","last_peered":"2024-04-25T19:12:45.692024+0000","last_clean":"2024-04-25T19:12:45.692024+0000","last_became_active":"2024-04-25T19:12:44.706268+0000","last_became_peered":"2024-04-25T19:12:44.706268+0000","last_unstale":"2024-04-25T19:12:45.692024+0000","last_undegraded":"2024-04-25T19:12:45.692024+0000","last_fullsized":"2024-04-25T19:12:45.692024+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:21.849594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691487+0000","last_change":"2024-04-25T19:12:44.682967+0000","last_active":"2024-04-25T19:12:45.691487+0000","last_peered":"2024-04-25T19:12:45.691487+0000","last_clean":"2024-04-25T19:12:45.691487+0000","last_became_active":"2024-04-25T19:12:44.682477+0000","last_became_peered":"2024-04-25T19:12:44.682477+0000","last_unstale":"2024-04-25T19:12:45.691487+0000","last_undegraded":"2024-04-25T19:12:45.691487+0000","last_fullsized":"2024-04-25T19:12:45.691487+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:45.225366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692065+0000","last_change":"2024-04-25T19:12:44.699586+0000","last_active":"2024-04-25T19:12:45.692065+0000","last_peered":"2024-04-25T19:12:45.692065+0000","last_clean":"2024-04-25T19:12:45.692065+0000","last_became_active":"2024-04-25T19:12:44.699289+0000","last_became_peered":"2024-04-25T19:12:44.699289+0000","last_unstale":"2024-04-25T19:12:45.692065+0000","last_undegraded":"2024-04-25T19:12:45.692065+0000","last_fullsized":"2024-04-25T19:12:45.692065+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:22.202361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691741+0000","last_change":"2024-04-25T19:12:44.702160+0000","last_active":"2024-04-25T19:12:45.691741+0000","last_peered":"2024-04-25T19:12:45.691741+0000","last_clean":"2024-04-25T19:12:45.691741+0000","last_became_active":"2024-04-25T19:12:44.701815+0000","last_became_peered":"2024-04-25T19:12:44.701815+0000","last_unstale":"2024-04-25T19:12:45.691741+0000","last_undegraded":"2024-04-25T19:12:45.691741+0000","last_fullsized":"2024-04-25T19:12:45.691741+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:45.995608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.692617+0000","last_change":"2024-04-25T19:12:44.691655+0000","last_active":"2024-04-25T19:12:44.692617+0000","last_peered":"2024-04-25T19:12:44.692617+0000","last_clean":"2024-04-25T19:12:44.692617+0000","last_became_active":"2024-04-25T19:12:44.691354+0000","last_became_peered":"2024-04-25T19:12:44.691354+0000","last_unstale":"2024-04-25T19:12:44.692617+0000","last_undegraded":"2024-04-25T19:12:44.692617+0000","last_fullsized":"2024-04-25T19:12:44.692617+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:00.405188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691421+0000","last_change":"2024-04-25T19:12:44.705515+0000","last_active":"2024-04-25T19:12:45.691421+0000","last_peered":"2024-04-25T19:12:45.691421+0000","last_clean":"2024-04-25T19:12:45.691421+0000","last_became_active":"2024-04-25T19:12:44.703843+0000","last_became_peered":"2024-04-25T19:12:44.703843+0000","last_unstale":"2024-04-25T19:12:45.691421+0000","last_undegraded":"2024-04-25T19:12:45.691421+0000","last_fullsized":"2024-04-25T19:12:45.691421+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:28.034410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691329+0000","last_change":"2024-04-25T19:12:43.687421+0000","last_active":"2024-04-25T19:12:45.691329+0000","last_peered":"2024-04-25T19:12:45.691329+0000","last_clean":"2024-04-25T19:12:45.691329+0000","last_became_active":"2024-04-25T19:12:43.687150+0000","last_became_peered":"2024-04-25T19:12:43.687150+0000","last_unstale":"2024-04-25T19:12:45.691329+0000","last_undegraded":"2024-04-25T19:12:45.691329+0000","last_fullsized":"2024-04-25T19:12:45.691329+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:22.014467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702763+0000","last_change":"2024-04-25T19:12:44.698702+0000","last_active":"2024-04-25T19:12:44.702763+0000","last_peered":"2024-04-25T19:12:44.702763+0000","last_clean":"2024-04-25T19:12:44.702763+0000","last_became_active":"2024-04-25T19:12:44.698378+0000","last_became_peered":"2024-04-25T19:12:44.698378+0000","last_unstale":"2024-04-25T19:12:44.702763+0000","last_undegraded":"2024-04-25T19:12:44.702763+0000","last_fullsized":"2024-04-25T19:12:44.702763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:37.425520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.708263+0000","last_change":"2024-04-25T19:12:43.687002+0000","last_active":"2024-04-25T19:12:44.708263+0000","last_peered":"2024-04-25T19:12:44.708263+0000","last_clean":"2024-04-25T19:12:44.708263+0000","last_became_active":"2024-04-25T19:12:43.686681+0000","last_became_peered":"2024-04-25T19:12:43.686681+0000","last_unstale":"2024-04-25T19:12:44.708263+0000","last_undegraded":"2024-04-25T19:12:44.708263+0000","last_fullsized":"2024-04-25T19:12:44.708263+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.690406+0000","last_change":"2024-04-25T19:12:44.688116+0000","last_active":"2024-04-25T19:12:44.690406+0000","last_peered":"2024-04-25T19:12:44.690406+0000","last_clean":"2024-04-25T19:12:44.690406+0000","last_became_active":"2024-04-25T19:12:44.687776+0000","last_became_peered":"2024-04-25T19:12:44.687776+0000","last_unstale":"2024-04-25T19:12:44.690406+0000","last_undegraded":"2024-04-25T19:12:44.690406+0000","last_fullsized":"2024-04-25T19:12:44.690406+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:32:51.909308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.701918+0000","last_change":"2024-04-25T19:12:43.682264+0000","last_active":"2024-04-25T19:12:44.701918+0000","last_peered":"2024-04-25T19:12:44.701918+0000","last_clean":"2024-04-25T19:12:44.701918+0000","last_became_active":"2024-04-25T19:12:43.681918+0000","last_became_peered":"2024-04-25T19:12:43.681918+0000","last_unstale":"2024-04-25T19:12:44.701918+0000","last_undegraded":"2024-04-25T19:12:44.701918+0000","last_fullsized":"2024-04-25T19:12:44.701918+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:55.257918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702148+0000","last_change":"2024-04-25T19:12:44.699758+0000","last_active":"2024-04-25T19:12:44.702148+0000","last_peered":"2024-04-25T19:12:44.702148+0000","last_clean":"2024-04-25T19:12:44.702148+0000","last_became_active":"2024-04-25T19:12:44.699294+0000","last_became_peered":"2024-04-25T19:12:44.699294+0000","last_unstale":"2024-04-25T19:12:44.702148+0000","last_undegraded":"2024-04-25T19:12:44.702148+0000","last_fullsized":"2024-04-25T19:12:44.702148+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:45.995608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.708446+0000","last_change":"2024-04-25T19:12:43.690606+0000","last_active":"2024-04-25T19:12:44.708446+0000","last_peered":"2024-04-25T19:12:44.708446+0000","last_clean":"2024-04-25T19:12:44.708446+0000","last_became_active":"2024-04-25T19:12:43.690163+0000","last_became_peered":"2024-04-25T19:12:43.690163+0000","last_unstale":"2024-04-25T19:12:44.708446+0000","last_undegraded":"2024-04-25T19:12:44.708446+0000","last_fullsized":"2024-04-25T19:12:44.708446+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:44.282277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.696212+0000","last_change":"2024-04-25T19:12:44.692946+0000","last_active":"2024-04-25T19:12:44.696212+0000","last_peered":"2024-04-25T19:12:44.696212+0000","last_clean":"2024-04-25T19:12:44.696212+0000","last_became_active":"2024-04-25T19:12:44.692070+0000","last_became_peered":"2024-04-25T19:12:44.692070+0000","last_unstale":"2024-04-25T19:12:44.696212+0000","last_undegraded":"2024-04-25T19:12:44.696212+0000","last_fullsized":"2024-04-25T19:12:44.696212+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:37.425520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691647+0000","last_change":"2024-04-25T19:12:43.682355+0000","last_active":"2024-04-25T19:12:45.691647+0000","last_peered":"2024-04-25T19:12:45.691647+0000","last_clean":"2024-04-25T19:12:45.691647+0000","last_became_active":"2024-04-25T19:12:43.682041+0000","last_became_peered":"2024-04-25T19:12:43.682041+0000","last_unstale":"2024-04-25T19:12:45.691647+0000","last_undegraded":"2024-04-25T19:12:45.691647+0000","last_fullsized":"2024-04-25T19:12:45.691647+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:44.218685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690685+0000","last_change":"2024-04-25T19:12:44.691213+0000","last_active":"2024-04-25T19:12:45.690685+0000","last_peered":"2024-04-25T19:12:45.690685+0000","last_clean":"2024-04-25T19:12:45.690685+0000","last_became_active":"2024-04-25T19:12:44.690834+0000","last_became_peered":"2024-04-25T19:12:44.690834+0000","last_unstale":"2024-04-25T19:12:45.690685+0000","last_undegraded":"2024-04-25T19:12:45.690685+0000","last_fullsized":"2024-04-25T19:12:45.690685+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:17.267296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690316+0000","last_change":"2024-04-25T19:12:43.677641+0000","last_active":"2024-04-25T19:12:45.690316+0000","last_peered":"2024-04-25T19:12:45.690316+0000","last_clean":"2024-04-25T19:12:45.690316+0000","last_became_active":"2024-04-25T19:12:43.677313+0000","last_became_peered":"2024-04-25T19:12:43.677313+0000","last_unstale":"2024-04-25T19:12:45.690316+0000","last_undegraded":"2024-04-25T19:12:45.690316+0000","last_fullsized":"2024-04-25T19:12:45.690316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:26.370845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.700978+0000","last_change":"2024-04-25T19:12:44.694711+0000","last_active":"2024-04-25T19:12:44.700978+0000","last_peered":"2024-04-25T19:12:44.700978+0000","last_clean":"2024-04-25T19:12:44.700978+0000","last_became_active":"2024-04-25T19:12:44.693493+0000","last_became_peered":"2024-04-25T19:12:44.693493+0000","last_unstale":"2024-04-25T19:12:44.700978+0000","last_undegraded":"2024-04-25T19:12:44.700978+0000","last_fullsized":"2024-04-25T19:12:44.700978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:54.655714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.678419+0000","last_change":"2024-04-25T19:12:43.691781+0000","last_active":"2024-04-25T19:12:44.678419+0000","last_peered":"2024-04-25T19:12:44.678419+0000","last_clean":"2024-04-25T19:12:44.678419+0000","last_became_active":"2024-04-25T19:12:43.691468+0000","last_became_peered":"2024-04-25T19:12:43.691468+0000","last_unstale":"2024-04-25T19:12:44.678419+0000","last_undegraded":"2024-04-25T19:12:44.678419+0000","last_fullsized":"2024-04-25T19:12:44.678419+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:26.370845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.706217+0000","last_change":"2024-04-25T19:12:44.703258+0000","last_active":"2024-04-25T19:12:44.706217+0000","last_peered":"2024-04-25T19:12:44.706217+0000","last_clean":"2024-04-25T19:12:44.706217+0000","last_became_active":"2024-04-25T19:12:44.702920+0000","last_became_peered":"2024-04-25T19:12:44.702920+0000","last_unstale":"2024-04-25T19:12:44.706217+0000","last_undegraded":"2024-04-25T19:12:44.706217+0000","last_fullsized":"2024-04-25T19:12:44.706217+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.684150+0000","last_change":"2024-04-25T19:12:43.687531+0000","last_active":"2024-04-25T19:12:44.684150+0000","last_peered":"2024-04-25T19:12:44.684150+0000","last_clean":"2024-04-25T19:12:44.684150+0000","last_became_active":"2024-04-25T19:12:43.687230+0000","last_became_peered":"2024-04-25T19:12:43.687230+0000","last_unstale":"2024-04-25T19:12:44.684150+0000","last_undegraded":"2024-04-25T19:12:44.684150+0000","last_fullsized":"2024-04-25T19:12:44.684150+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:22.014467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.685624+0000","last_change":"2024-04-25T19:12:44.684353+0000","last_active":"2024-04-25T19:12:44.685624+0000","last_peered":"2024-04-25T19:12:44.685624+0000","last_clean":"2024-04-25T19:12:44.685624+0000","last_became_active":"2024-04-25T19:12:44.684024+0000","last_became_peered":"2024-04-25T19:12:44.684024+0000","last_unstale":"2024-04-25T19:12:44.685624+0000","last_undegraded":"2024-04-25T19:12:44.685624+0000","last_fullsized":"2024-04-25T19:12:44.685624+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:35.384793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.686423+0000","last_change":"2024-04-25T19:12:43.685597+0000","last_active":"2024-04-25T19:12:44.686423+0000","last_peered":"2024-04-25T19:12:44.686423+0000","last_clean":"2024-04-25T19:12:44.686423+0000","last_became_active":"2024-04-25T19:12:43.685245+0000","last_became_peered":"2024-04-25T19:12:43.685245+0000","last_unstale":"2024-04-25T19:12:44.686423+0000","last_undegraded":"2024-04-25T19:12:44.686423+0000","last_fullsized":"2024-04-25T19:12:44.686423+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:35.845516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692014+0000","last_change":"2024-04-25T19:12:44.698686+0000","last_active":"2024-04-25T19:12:45.692014+0000","last_peered":"2024-04-25T19:12:45.692014+0000","last_clean":"2024-04-25T19:12:45.692014+0000","last_became_active":"2024-04-25T19:12:44.697877+0000","last_became_peered":"2024-04-25T19:12:44.697877+0000","last_unstale":"2024-04-25T19:12:45.692014+0000","last_undegraded":"2024-04-25T19:12:45.692014+0000","last_fullsized":"2024-04-25T19:12:45.692014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:29.311080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.693689+0000","last_change":"2024-04-25T19:12:43.691155+0000","last_active":"2024-04-25T19:12:45.693689+0000","last_peered":"2024-04-25T19:12:45.693689+0000","last_clean":"2024-04-25T19:12:45.693689+0000","last_became_active":"2024-04-25T19:12:43.690735+0000","last_became_peered":"2024-04-25T19:12:43.690735+0000","last_unstale":"2024-04-25T19:12:45.693689+0000","last_undegraded":"2024-04-25T19:12:45.693689+0000","last_fullsized":"2024-04-25T19:12:45.693689+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:36.418839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.698998+0000","last_change":"2024-04-25T19:12:44.698058+0000","last_active":"2024-04-25T19:12:44.698998+0000","last_peered":"2024-04-25T19:12:44.698998+0000","last_clean":"2024-04-25T19:12:44.698998+0000","last_became_active":"2024-04-25T19:12:44.695384+0000","last_became_peered":"2024-04-25T19:12:44.695384+0000","last_unstale":"2024-04-25T19:12:44.698998+0000","last_undegraded":"2024-04-25T19:12:44.698998+0000","last_fullsized":"2024-04-25T19:12:44.698998+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:08:46.997852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692326+0000","last_change":"2024-04-25T19:12:43.686664+0000","last_active":"2024-04-25T19:12:45.692326+0000","last_peered":"2024-04-25T19:12:45.692326+0000","last_clean":"2024-04-25T19:12:45.692326+0000","last_became_active":"2024-04-25T19:12:43.686375+0000","last_became_peered":"2024-04-25T19:12:43.686375+0000","last_unstale":"2024-04-25T19:12:45.692326+0000","last_undegraded":"2024-04-25T19:12:45.692326+0000","last_fullsized":"2024-04-25T19:12:45.692326+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:35.845516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.691231+0000","last_change":"2024-04-25T19:12:44.690212+0000","last_active":"2024-04-25T19:12:44.691231+0000","last_peered":"2024-04-25T19:12:44.691231+0000","last_clean":"2024-04-25T19:12:44.691231+0000","last_became_active":"2024-04-25T19:12:44.689709+0000","last_became_peered":"2024-04-25T19:12:44.689709+0000","last_unstale":"2024-04-25T19:12:44.691231+0000","last_undegraded":"2024-04-25T19:12:44.691231+0000","last_fullsized":"2024-04-25T19:12:44.691231+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:27.377526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":18,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.710832+0000","last_change":"2024-04-25T19:12:43.689503+0000","last_active":"2024-04-25T19:12:44.710832+0000","last_peered":"2024-04-25T19:12:44.710832+0000","last_clean":"2024-04-25T19:12:44.710832+0000","last_became_active":"2024-04-25T19:12:43.689203+0000","last_became_peered":"2024-04-25T19:12:43.689203+0000","last_unstale":"2024-04-25T19:12:44.710832+0000","last_undegraded":"2024-04-25T19:12:44.710832+0000","last_fullsized":"2024-04-25T19:12:44.710832+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:08:45.991172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.696291+0000","last_change":"2024-04-25T19:12:44.693888+0000","last_active":"2024-04-25T19:12:44.696291+0000","last_peered":"2024-04-25T19:12:44.696291+0000","last_clean":"2024-04-25T19:12:44.696291+0000","last_became_active":"2024-04-25T19:12:44.691696+0000","last_became_peered":"2024-04-25T19:12:44.691696+0000","last_unstale":"2024-04-25T19:12:44.696291+0000","last_undegraded":"2024-04-25T19:12:44.696291+0000","last_fullsized":"2024-04-25T19:12:44.696291+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:17.038220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.701994+0000","last_change":"2024-04-25T19:12:43.691713+0000","last_active":"2024-04-25T19:12:44.701994+0000","last_peered":"2024-04-25T19:12:44.701994+0000","last_clean":"2024-04-25T19:12:44.701994+0000","last_became_active":"2024-04-25T19:12:43.691375+0000","last_became_peered":"2024-04-25T19:12:43.691375+0000","last_unstale":"2024-04-25T19:12:44.701994+0000","last_undegraded":"2024-04-25T19:12:44.701994+0000","last_fullsized":"2024-04-25T19:12:44.701994+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:58.179259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702082+0000","last_change":"2024-04-25T19:12:44.699213+0000","last_active":"2024-04-25T19:12:44.702082+0000","last_peered":"2024-04-25T19:12:44.702082+0000","last_clean":"2024-04-25T19:12:44.702082+0000","last_became_active":"2024-04-25T19:12:44.698893+0000","last_became_peered":"2024-04-25T19:12:44.698893+0000","last_unstale":"2024-04-25T19:12:44.702082+0000","last_undegraded":"2024-04-25T19:12:44.702082+0000","last_fullsized":"2024-04-25T19:12:44.702082+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:54.272820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.678812+0000","last_change":"2024-04-25T19:12:43.691127+0000","last_active":"2024-04-25T19:12:44.678812+0000","last_peered":"2024-04-25T19:12:44.678812+0000","last_clean":"2024-04-25T19:12:44.678812+0000","last_became_active":"2024-04-25T19:12:43.690372+0000","last_became_peered":"2024-04-25T19:12:43.690372+0000","last_unstale":"2024-04-25T19:12:44.678812+0000","last_undegraded":"2024-04-25T19:12:44.678812+0000","last_fullsized":"2024-04-25T19:12:44.678812+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.690328+0000","last_change":"2024-04-25T19:12:44.698557+0000","last_active":"2024-04-25T19:12:45.690328+0000","last_peered":"2024-04-25T19:12:45.690328+0000","last_clean":"2024-04-25T19:12:45.690328+0000","last_became_active":"2024-04-25T19:12:44.698170+0000","last_became_peered":"2024-04-25T19:12:44.698170+0000","last_unstale":"2024-04-25T19:12:45.690328+0000","last_undegraded":"2024-04-25T19:12:45.690328+0000","last_fullsized":"2024-04-25T19:12:45.690328+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:20:58.680446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692316+0000","last_change":"2024-04-25T19:12:43.687570+0000","last_active":"2024-04-25T19:12:45.692316+0000","last_peered":"2024-04-25T19:12:45.692316+0000","last_clean":"2024-04-25T19:12:45.692316+0000","last_became_active":"2024-04-25T19:12:43.687216+0000","last_became_peered":"2024-04-25T19:12:43.687216+0000","last_unstale":"2024-04-25T19:12:45.692316+0000","last_undegraded":"2024-04-25T19:12:45.692316+0000","last_fullsized":"2024-04-25T19:12:45.692316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:56.285928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702120+0000","last_change":"2024-04-25T19:12:44.699754+0000","last_active":"2024-04-25T19:12:44.702120+0000","last_peered":"2024-04-25T19:12:44.702120+0000","last_clean":"2024-04-25T19:12:44.702120+0000","last_became_active":"2024-04-25T19:12:44.699422+0000","last_became_peered":"2024-04-25T19:12:44.699422+0000","last_unstale":"2024-04-25T19:12:44.702120+0000","last_undegraded":"2024-04-25T19:12:44.702120+0000","last_fullsized":"2024-04-25T19:12:44.702120+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:05.288431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692389+0000","last_change":"2024-04-25T19:12:43.677420+0000","last_active":"2024-04-25T19:12:45.692389+0000","last_peered":"2024-04-25T19:12:45.692389+0000","last_clean":"2024-04-25T19:12:45.692389+0000","last_became_active":"2024-04-25T19:12:43.677108+0000","last_became_peered":"2024-04-25T19:12:43.677108+0000","last_unstale":"2024-04-25T19:12:45.692389+0000","last_undegraded":"2024-04-25T19:12:45.692389+0000","last_fullsized":"2024-04-25T19:12:45.692389+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:32:50.902627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702208+0000","last_change":"2024-04-25T19:12:44.699717+0000","last_active":"2024-04-25T19:12:44.702208+0000","last_peered":"2024-04-25T19:12:44.702208+0000","last_clean":"2024-04-25T19:12:44.702208+0000","last_became_active":"2024-04-25T19:12:44.699361+0000","last_became_peered":"2024-04-25T19:12:44.699361+0000","last_unstale":"2024-04-25T19:12:44.702208+0000","last_undegraded":"2024-04-25T19:12:44.702208+0000","last_fullsized":"2024-04-25T19:12:44.702208+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:28.034410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692338+0000","last_change":"2024-04-25T19:12:43.679044+0000","last_active":"2024-04-25T19:12:45.692338+0000","last_peered":"2024-04-25T19:12:45.692338+0000","last_clean":"2024-04-25T19:12:45.692338+0000","last_became_active":"2024-04-25T19:12:43.678723+0000","last_became_peered":"2024-04-25T19:12:43.678723+0000","last_unstale":"2024-04-25T19:12:45.692338+0000","last_undegraded":"2024-04-25T19:12:45.692338+0000","last_fullsized":"2024-04-25T19:12:45.692338+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:16.031539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692310+0000","last_change":"2024-04-25T19:12:44.697124+0000","last_active":"2024-04-25T19:12:45.692310+0000","last_peered":"2024-04-25T19:12:45.692310+0000","last_clean":"2024-04-25T19:12:45.692310+0000","last_became_active":"2024-04-25T19:12:44.695351+0000","last_became_peered":"2024-04-25T19:12:44.695351+0000","last_unstale":"2024-04-25T19:12:45.692310+0000","last_undegraded":"2024-04-25T19:12:45.692310+0000","last_fullsized":"2024-04-25T19:12:45.692310+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:24.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.697383+0000","last_change":"2024-04-25T19:12:43.678051+0000","last_active":"2024-04-25T19:12:44.697383+0000","last_peered":"2024-04-25T19:12:44.697383+0000","last_clean":"2024-04-25T19:12:44.697383+0000","last_became_active":"2024-04-25T19:12:43.677640+0000","last_became_peered":"2024-04-25T19:12:43.677640+0000","last_unstale":"2024-04-25T19:12:44.697383+0000","last_undegraded":"2024-04-25T19:12:44.697383+0000","last_fullsized":"2024-04-25T19:12:44.697383+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:06.360629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.700904+0000","last_change":"2024-04-25T19:12:44.695748+0000","last_active":"2024-04-25T19:12:44.700904+0000","last_peered":"2024-04-25T19:12:44.700904+0000","last_clean":"2024-04-25T19:12:44.700904+0000","last_became_active":"2024-04-25T19:12:44.695447+0000","last_became_peered":"2024-04-25T19:12:44.695447+0000","last_unstale":"2024-04-25T19:12:44.700904+0000","last_undegraded":"2024-04-25T19:12:44.700904+0000","last_fullsized":"2024-04-25T19:12:44.700904+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:34.463674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.708696+0000","last_change":"2024-04-25T19:12:43.684026+0000","last_active":"2024-04-25T19:12:44.708696+0000","last_peered":"2024-04-25T19:12:44.708696+0000","last_clean":"2024-04-25T19:12:44.708696+0000","last_became_active":"2024-04-25T19:12:43.683693+0000","last_became_peered":"2024-04-25T19:12:43.683693+0000","last_unstale":"2024-04-25T19:12:44.708696+0000","last_undegraded":"2024-04-25T19:12:44.708696+0000","last_fullsized":"2024-04-25T19:12:44.708696+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:14.924183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.703516+0000","last_change":"2024-04-25T19:12:44.701567+0000","last_active":"2024-04-25T19:12:44.703516+0000","last_peered":"2024-04-25T19:12:44.703516+0000","last_clean":"2024-04-25T19:12:44.703516+0000","last_became_active":"2024-04-25T19:12:44.701201+0000","last_became_peered":"2024-04-25T19:12:44.701201+0000","last_unstale":"2024-04-25T19:12:44.703516+0000","last_undegraded":"2024-04-25T19:12:44.703516+0000","last_fullsized":"2024-04-25T19:12:44.703516+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:27.013579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.679214+0000","last_change":"2024-04-25T19:12:43.684026+0000","last_active":"2024-04-25T19:12:44.679214+0000","last_peered":"2024-04-25T19:12:44.679214+0000","last_clean":"2024-04-25T19:12:44.679214+0000","last_became_active":"2024-04-25T19:12:43.683700+0000","last_became_peered":"2024-04-25T19:12:43.683700+0000","last_unstale":"2024-04-25T19:12:44.679214+0000","last_undegraded":"2024-04-25T19:12:44.679214+0000","last_fullsized":"2024-04-25T19:12:44.679214+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:33.456993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692569+0000","last_change":"2024-04-25T19:12:44.699373+0000","last_active":"2024-04-25T19:12:45.692569+0000","last_peered":"2024-04-25T19:12:45.692569+0000","last_clean":"2024-04-25T19:12:45.692569+0000","last_became_active":"2024-04-25T19:12:44.693763+0000","last_became_peered":"2024-04-25T19:12:44.693763+0000","last_unstale":"2024-04-25T19:12:45.692569+0000","last_undegraded":"2024-04-25T19:12:45.692569+0000","last_fullsized":"2024-04-25T19:12:45.692569+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.701058+0000","last_change":"2024-04-25T19:12:43.687714+0000","last_active":"2024-04-25T19:12:44.701058+0000","last_peered":"2024-04-25T19:12:44.701058+0000","last_clean":"2024-04-25T19:12:44.701058+0000","last_became_active":"2024-04-25T19:12:43.687375+0000","last_became_peered":"2024-04-25T19:12:43.687375+0000","last_unstale":"2024-04-25T19:12:44.701058+0000","last_undegraded":"2024-04-25T19:12:44.701058+0000","last_fullsized":"2024-04-25T19:12:44.701058+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:01.568678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.696878+0000","last_change":"2024-04-25T19:12:44.694953+0000","last_active":"2024-04-25T19:12:44.696878+0000","last_peered":"2024-04-25T19:12:44.696878+0000","last_clean":"2024-04-25T19:12:44.696878+0000","last_became_active":"2024-04-25T19:12:44.694607+0000","last_became_peered":"2024-04-25T19:12:44.694607+0000","last_unstale":"2024-04-25T19:12:44.696878+0000","last_undegraded":"2024-04-25T19:12:44.696878+0000","last_fullsized":"2024-04-25T19:12:44.696878+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:36.852197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.671698+0000","last_change":"2024-04-25T19:12:43.688266+0000","last_active":"2024-04-25T19:12:44.671698+0000","last_peered":"2024-04-25T19:12:44.671698+0000","last_clean":"2024-04-25T19:12:44.671698+0000","last_became_active":"2024-04-25T19:12:43.687912+0000","last_became_peered":"2024-04-25T19:12:43.687912+0000","last_unstale":"2024-04-25T19:12:44.671698+0000","last_undegraded":"2024-04-25T19:12:44.671698+0000","last_fullsized":"2024-04-25T19:12:44.671698+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:54.337918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691090+0000","last_change":"2024-04-25T19:12:44.687324+0000","last_active":"2024-04-25T19:12:45.691090+0000","last_peered":"2024-04-25T19:12:45.691090+0000","last_clean":"2024-04-25T19:12:45.691090+0000","last_became_active":"2024-04-25T19:12:44.686274+0000","last_became_peered":"2024-04-25T19:12:44.686274+0000","last_unstale":"2024-04-25T19:12:45.691090+0000","last_undegraded":"2024-04-25T19:12:45.691090+0000","last_fullsized":"2024-04-25T19:12:45.691090+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:20.222641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691774+0000","last_change":"2024-04-25T19:12:43.687265+0000","last_active":"2024-04-25T19:12:45.691774+0000","last_peered":"2024-04-25T19:12:45.691774+0000","last_clean":"2024-04-25T19:12:45.691774+0000","last_became_active":"2024-04-25T19:12:43.686948+0000","last_became_peered":"2024-04-25T19:12:43.686948+0000","last_unstale":"2024-04-25T19:12:45.691774+0000","last_undegraded":"2024-04-25T19:12:45.691774+0000","last_fullsized":"2024-04-25T19:12:45.691774+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:55.257918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.706746+0000","last_change":"2024-04-25T19:12:44.706030+0000","last_active":"2024-04-25T19:12:44.706746+0000","last_peered":"2024-04-25T19:12:44.706746+0000","last_clean":"2024-04-25T19:12:44.706746+0000","last_became_active":"2024-04-25T19:12:44.704826+0000","last_became_peered":"2024-04-25T19:12:44.704826+0000","last_unstale":"2024-04-25T19:12:44.706746+0000","last_undegraded":"2024-04-25T19:12:44.706746+0000","last_fullsized":"2024-04-25T19:12:44.706746+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.671506+0000","last_change":"2024-04-25T19:12:43.688066+0000","last_active":"2024-04-25T19:12:44.671506+0000","last_peered":"2024-04-25T19:12:44.671506+0000","last_clean":"2024-04-25T19:12:44.671506+0000","last_became_active":"2024-04-25T19:12:43.687694+0000","last_became_peered":"2024-04-25T19:12:43.687694+0000","last_unstale":"2024-04-25T19:12:44.671506+0000","last_undegraded":"2024-04-25T19:12:44.671506+0000","last_fullsized":"2024-04-25T19:12:44.671506+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:24.803044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.702122+0000","last_change":"2024-04-25T19:12:44.698861+0000","last_active":"2024-04-25T19:12:44.702122+0000","last_peered":"2024-04-25T19:12:44.702122+0000","last_clean":"2024-04-25T19:12:44.702122+0000","last_became_active":"2024-04-25T19:12:44.698178+0000","last_became_peered":"2024-04-25T19:12:44.698178+0000","last_unstale":"2024-04-25T19:12:44.702122+0000","last_undegraded":"2024-04-25T19:12:44.702122+0000","last_fullsized":"2024-04-25T19:12:44.702122+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:22.202361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692565+0000","last_change":"2024-04-25T19:12:43.685799+0000","last_active":"2024-04-25T19:12:45.692565+0000","last_peered":"2024-04-25T19:12:45.692565+0000","last_clean":"2024-04-25T19:12:45.692565+0000","last_became_active":"2024-04-25T19:12:43.685478+0000","last_became_peered":"2024-04-25T19:12:43.685478+0000","last_unstale":"2024-04-25T19:12:45.692565+0000","last_undegraded":"2024-04-25T19:12:45.692565+0000","last_fullsized":"2024-04-25T19:12:45.692565+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:21.195681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692175+0000","last_change":"2024-04-25T19:12:44.699033+0000","last_active":"2024-04-25T19:12:45.692175+0000","last_peered":"2024-04-25T19:12:45.692175+0000","last_clean":"2024-04-25T19:12:45.692175+0000","last_became_active":"2024-04-25T19:12:44.698376+0000","last_became_peered":"2024-04-25T19:12:44.698376+0000","last_unstale":"2024-04-25T19:12:45.692175+0000","last_undegraded":"2024-04-25T19:12:45.692175+0000","last_fullsized":"2024-04-25T19:12:45.692175+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:57.582503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.672020+0000","last_change":"2024-04-25T19:12:43.688186+0000","last_active":"2024-04-25T19:12:44.672020+0000","last_peered":"2024-04-25T19:12:44.672020+0000","last_clean":"2024-04-25T19:12:44.672020+0000","last_became_active":"2024-04-25T19:12:43.687361+0000","last_became_peered":"2024-04-25T19:12:43.687361+0000","last_unstale":"2024-04-25T19:12:44.672020+0000","last_undegraded":"2024-04-25T19:12:44.672020+0000","last_fullsized":"2024-04-25T19:12:44.672020+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:37.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.706462+0000","last_change":"2024-04-25T19:12:44.705586+0000","last_active":"2024-04-25T19:12:44.706462+0000","last_peered":"2024-04-25T19:12:44.706462+0000","last_clean":"2024-04-25T19:12:44.706462+0000","last_became_active":"2024-04-25T19:12:44.704667+0000","last_became_peered":"2024-04-25T19:12:44.704667+0000","last_unstale":"2024-04-25T19:12:44.706462+0000","last_undegraded":"2024-04-25T19:12:44.706462+0000","last_fullsized":"2024-04-25T19:12:44.706462+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:12.662654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.678072+0000","last_change":"2024-04-25T19:12:43.685345+0000","last_active":"2024-04-25T19:12:44.678072+0000","last_peered":"2024-04-25T19:12:44.678072+0000","last_clean":"2024-04-25T19:12:44.678072+0000","last_became_active":"2024-04-25T19:12:43.684968+0000","last_became_peered":"2024-04-25T19:12:43.684968+0000","last_unstale":"2024-04-25T19:12:44.678072+0000","last_undegraded":"2024-04-25T19:12:44.678072+0000","last_fullsized":"2024-04-25T19:12:44.678072+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:33.456993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.704151+0000","last_change":"2024-04-25T19:12:44.702919+0000","last_active":"2024-04-25T19:12:44.704151+0000","last_peered":"2024-04-25T19:12:44.704151+0000","last_clean":"2024-04-25T19:12:44.704151+0000","last_became_active":"2024-04-25T19:12:44.702600+0000","last_became_peered":"2024-04-25T19:12:44.702600+0000","last_unstale":"2024-04-25T19:12:44.704151+0000","last_undegraded":"2024-04-25T19:12:44.704151+0000","last_fullsized":"2024-04-25T19:12:44.704151+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:59.185940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.678903+0000","last_change":"2024-04-25T19:12:43.689072+0000","last_active":"2024-04-25T19:12:44.678903+0000","last_peered":"2024-04-25T19:12:44.678903+0000","last_clean":"2024-04-25T19:12:44.678903+0000","last_became_active":"2024-04-25T19:12:43.688665+0000","last_became_peered":"2024-04-25T19:12:43.688665+0000","last_unstale":"2024-04-25T19:12:44.678903+0000","last_undegraded":"2024-04-25T19:12:44.678903+0000","last_fullsized":"2024-04-25T19:12:44.678903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:23.111993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.689342+0000","last_change":"2024-04-25T19:12:44.683442+0000","last_active":"2024-04-25T19:12:44.689342+0000","last_peered":"2024-04-25T19:12:44.689342+0000","last_clean":"2024-04-25T19:12:44.689342+0000","last_became_active":"2024-04-25T19:12:44.683137+0000","last_became_peered":"2024-04-25T19:12:44.683137+0000","last_unstale":"2024-04-25T19:12:44.689342+0000","last_undegraded":"2024-04-25T19:12:44.689342+0000","last_fullsized":"2024-04-25T19:12:44.689342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:24.118674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692192+0000","last_change":"2024-04-25T19:12:43.688566+0000","last_active":"2024-04-25T19:12:45.692192+0000","last_peered":"2024-04-25T19:12:45.692192+0000","last_clean":"2024-04-25T19:12:45.692192+0000","last_became_active":"2024-04-25T19:12:43.688009+0000","last_became_peered":"2024-04-25T19:12:43.688009+0000","last_unstale":"2024-04-25T19:12:45.692192+0000","last_undegraded":"2024-04-25T19:12:45.692192+0000","last_fullsized":"2024-04-25T19:12:45.692192+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:47.259986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.708536+0000","last_change":"2024-04-25T19:12:44.704959+0000","last_active":"2024-04-25T19:12:44.708536+0000","last_peered":"2024-04-25T19:12:44.708536+0000","last_clean":"2024-04-25T19:12:44.708536+0000","last_became_active":"2024-04-25T19:12:44.704200+0000","last_became_peered":"2024-04-25T19:12:44.704200+0000","last_unstale":"2024-04-25T19:12:44.708536+0000","last_undegraded":"2024-04-25T19:12:44.708536+0000","last_fullsized":"2024-04-25T19:12:44.708536+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:24.118674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.684579+0000","last_change":"2024-04-25T19:12:43.686166+0000","last_active":"2024-04-25T19:12:44.684579+0000","last_peered":"2024-04-25T19:12:44.684579+0000","last_clean":"2024-04-25T19:12:44.684579+0000","last_became_active":"2024-04-25T19:12:43.685724+0000","last_became_peered":"2024-04-25T19:12:43.685724+0000","last_unstale":"2024-04-25T19:12:44.684579+0000","last_undegraded":"2024-04-25T19:12:44.684579+0000","last_fullsized":"2024-04-25T19:12:44.684579+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:00:37.087793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691109+0000","last_change":"2024-04-25T19:12:44.702272+0000","last_active":"2024-04-25T19:12:45.691109+0000","last_peered":"2024-04-25T19:12:45.691109+0000","last_clean":"2024-04-25T19:12:45.691109+0000","last_became_active":"2024-04-25T19:12:44.701778+0000","last_became_peered":"2024-04-25T19:12:44.701778+0000","last_unstale":"2024-04-25T19:12:45.691109+0000","last_undegraded":"2024-04-25T19:12:45.691109+0000","last_fullsized":"2024-04-25T19:12:45.691109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:54.272820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-25T19:12:44.693344+0000","last_change":"2024-04-25T19:12:43.686223+0000","last_active":"2024-04-25T19:12:44.693344+0000","last_peered":"2024-04-25T19:12:44.693344+0000","last_clean":"2024-04-25T19:12:44.693344+0000","last_became_active":"2024-04-25T19:12:43.685881+0000","last_became_peered":"2024-04-25T19:12:43.685881+0000","last_unstale":"2024-04-25T19:12:44.693344+0000","last_undegraded":"2024-04-25T19:12:44.693344+0000","last_fullsized":"2024-04-25T19:12:44.693344+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:08.114375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692651+0000","last_change":"2024-04-25T19:12:44.706605+0000","last_active":"2024-04-25T19:12:45.692651+0000","last_peered":"2024-04-25T19:12:45.692651+0000","last_clean":"2024-04-25T19:12:45.692651+0000","last_became_active":"2024-04-25T19:12:44.706192+0000","last_became_peered":"2024-04-25T19:12:44.706192+0000","last_unstale":"2024-04-25T19:12:45.692651+0000","last_undegraded":"2024-04-25T19:12:45.692651+0000","last_fullsized":"2024-04-25T19:12:45.692651+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:34.452188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.691542+0000","last_change":"2024-04-25T19:12:43.689029+0000","last_active":"2024-04-25T19:12:45.691542+0000","last_peered":"2024-04-25T19:12:45.691542+0000","last_clean":"2024-04-25T19:12:45.691542+0000","last_became_active":"2024-04-25T19:12:43.688728+0000","last_became_peered":"2024-04-25T19:12:43.688728+0000","last_unstale":"2024-04-25T19:12:45.691542+0000","last_undegraded":"2024-04-25T19:12:45.691542+0000","last_fullsized":"2024-04-25T19:12:45.691542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":141,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T19:12:45.692639+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:12:45.692639+0000","last_peered":"2024-04-25T19:12:45.692639+0000","last_clean":"2024-04-25T19:12:45.692639+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:12:45.692639+0000","last_undegraded":"2024-04-25T19:12:45.692639+0000","last_fullsized":"2024-04-25T19:12:45.692639+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1036288,"data_stored":775386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31600,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":430080,"data_stored":183918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31604,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340236,"statfs":{"total":96636764160,"available":96604401664,"internally_reserved":0,"allocated":434176,"data_stored":183448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31604,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340236,"statfs":{"total":96636764160,"available":96604401664,"internally_reserved":0,"allocated":434176,"data_stored":183940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":184982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31600,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":430080,"data_stored":183406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31600,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340240,"statfs":{"total":96636764160,"available":96604405760,"internally_reserved":0,"allocated":430080,"data_stored":183426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32184,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339656,"statfs":{"total":96636764160,"available":96603807744,"internally_reserved":0,"allocated":1024000,"data_stored":773738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:12:49.112 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T19:12:49.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T19:12:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.265+0000 7f117a63c640 1 Processor -- start 2024-04-25T19:12:49.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.265+0000 7f117a63c640 1 -- start start 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.266+0000 7f117a63c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11740d5140 0x7f11740b4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.266+0000 7f117a63c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11740d1100 con 0x7f11741647c0 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.266+0000 7f117a63c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11740d13e0 con 0x7f11740b7580 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.266+0000 7f117a63c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f1174164330 con 0x7f11740d5140 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.266+0000 7f11737fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f11741647c0 0x7f11740d08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39760/0 (socket says 172.21.15.28:39760) 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.266+0000 7f11737fe640 1 -- 172.21.15.28:0/3777411767 learned_addr learned my addr 172.21.15.28:0/3777411767 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.267+0000 7f1173fff640 1 --2- 172.21.15.28:0/3777411767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11740d5140 0x7f11740b4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.267+0000 7f1172ffd640 1 -- 172.21.15.28:0/3777411767 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 213355978 0 0) 0x7f11740d1100 con 0x7f11741647c0 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.267+0000 7f1172ffd640 1 -- 172.21.15.28:0/3777411767 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f115c003520 con 0x7f11741647c0 2024-04-25T19:12:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.267+0000 7f1172ffd640 1 -- 172.21.15.28:0/3777411767 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2796503481 0 0) 0x7f11740d13e0 con 0x7f11740b7580 2024-04-25T19:12:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.267+0000 7f1172ffd640 1 -- 172.21.15.28:0/3777411767 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11740d1100 con 0x7f11740b7580 2024-04-25T19:12:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.267+0000 7f1173fff640 1 -- 172.21.15.28:0/3777411767 >> v1:172.21.15.53:6789/0 conn(0x7f11740b7580 legacy=0x7f11740b7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.267+0000 7f1173fff640 1 -- 172.21.15.28:0/3777411767 >> v1:172.21.15.28:6789/0 conn(0x7f11741647c0 legacy=0x7f11740d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.267+0000 7f1173fff640 1 -- 172.21.15.28:0/3777411767 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1174163fc0 con 0x7f11740d5140 2024-04-25T19:12:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.268+0000 7f1173fff640 1 --2- 172.21.15.28:0/3777411767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11740d5140 0x7f11740b4d20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f11741644b0 tx=0x7f116402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=501cc24adfbde374 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.268+0000 7f1172ffd640 1 -- 172.21.15.28:0/3777411767 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164037070 con 0x7f11740d5140 2024-04-25T19:12:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.268+0000 7f1172ffd640 1 -- 172.21.15.28:0/3777411767 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1164032440 con 0x7f11740d5140 2024-04-25T19:12:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.268+0000 7f1172ffd640 1 -- 172.21.15.28:0/3777411767 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11640327f0 con 0x7f11740d5140 2024-04-25T19:12:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.269+0000 7f117a63c640 1 -- 172.21.15.28:0/3777411767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11740d5140 msgr2=0x7f11740b4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.269+0000 7f117a63c640 1 --2- 172.21.15.28:0/3777411767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11740d5140 0x7f11740b4d20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f11741644b0 tx=0x7f116402a1c0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.269+0000 7f117a63c640 1 -- 172.21.15.28:0/3777411767 shutdown_connections 2024-04-25T19:12:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.269+0000 7f117a63c640 1 --2- 172.21.15.28:0/3777411767 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11740d5140 0x7f11740b4d20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.269+0000 7f117a63c640 1 -- 172.21.15.28:0/3777411767 >> 172.21.15.28:0/3777411767 conn(0x7f11740d2760 msgr2=0x7f11740d2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.269+0000 7f117a63c640 1 -- 172.21.15.28:0/3777411767 shutdown_connections 2024-04-25T19:12:49.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.269+0000 7f117a63c640 1 -- 172.21.15.28:0/3777411767 wait complete. 2024-04-25T19:12:49.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.271+0000 7f117a63c640 1 Processor -- start 2024-04-25T19:12:49.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.271+0000 7f117a63c640 1 -- start start 2024-04-25T19:12:49.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f117a63c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11740b7580 0x7f1174160d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f117a63c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11740d5140 0x7f11741612b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f117a63c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11741647c0 0x7f117415afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f117a63c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11740b9150 con 0x7f11740d5140 2024-04-25T19:12:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f117a63c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f11740b8fd0 con 0x7f11740b7580 2024-04-25T19:12:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f117a63c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f11740b92d0 con 0x7f11741647c0 2024-04-25T19:12:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f11737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11740d5140 0x7f11741612b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f11737fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11740d5140 0x7f11741612b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54130/0 (socket says 172.21.15.28:54130) 2024-04-25T19:12:49.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.272+0000 7f11737fe640 1 -- 172.21.15.28:0/4151654865 learned_addr learned my addr 172.21.15.28:0/4151654865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f1173fff640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11740b7580 0x7f1174160d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f1178bd9640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11741647c0 0x7f117415afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f11737fe640 1 -- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11741647c0 msgr2=0x7f117415afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f11737fe640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11741647c0 0x7f117415afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f11737fe640 1 -- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11740b7580 msgr2=0x7f1174160d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f11737fe640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11740b7580 0x7f1174160d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f11737fe640 1 -- 172.21.15.28:0/4151654865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1174155d20 con 0x7f11740d5140 2024-04-25T19:12:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f1173fff640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11740b7580 0x7f1174160d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:12:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f11737fe640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11740d5140 0x7f11741612b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f117408eb10 tx=0x7f116800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f1170ff9640 1 -- 172.21.15.28:0/4151654865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1168017070 con 0x7f11740d5140 2024-04-25T19:12:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f1170ff9640 1 -- 172.21.15.28:0/4151654865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1168005ce0 con 0x7f11740d5140 2024-04-25T19:12:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f1170ff9640 1 -- 172.21.15.28:0/4151654865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11680072f0 con 0x7f11740d5140 2024-04-25T19:12:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1174156000 con 0x7f11740d5140 2024-04-25T19:12:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.273+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11741565c0 con 0x7f11740d5140 2024-04-25T19:12:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.274+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1174050740 con 0x7f11740d5140 2024-04-25T19:12:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.275+0000 7f1170ff9640 1 -- 172.21.15.28:0/4151654865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f116801a070 con 0x7f11740d5140 2024-04-25T19:12:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.275+0000 7f1170ff9640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f11600744b0 0x7f11600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.275+0000 7f1170ff9640 1 -- 172.21.15.28:0/4151654865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f11680946e0 con 0x7f11740d5140 2024-04-25T19:12:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.275+0000 7f1173fff640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f11600744b0 0x7f11600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.276+0000 7f1173fff640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f11600744b0 0x7f11600769c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1164034000 tx=0x7f116402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:49.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.280+0000 7f1170ff9640 1 -- 172.21.15.28:0/4151654865 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f116805ed80 con 0x7f11740d5140 2024-04-25T19:12:49.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.536+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f11740bba20 con 0x7f11740d5140 2024-04-25T19:12:49.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.538+0000 7f1170ff9640 1 -- 172.21.15.28:0/4151654865 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f116805e680 con 0x7f11740d5140 2024-04-25T19:12:49.540 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-25T19:12:49.541 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQCqqipmM9YPOxAAJwNJw9dfCED4wy79hvmLNg== 2024-04-25T19:12:49.541 INFO:teuthology.orchestra.run.smithi028.stdout: caps mds = "allow" 2024-04-25T19:12:49.541 INFO:teuthology.orchestra.run.smithi028.stdout: caps mon = "allow r" 2024-04-25T19:12:49.541 INFO:teuthology.orchestra.run.smithi028.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T19:12:49.541 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-04-25T19:12:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f11600744b0 msgr2=0x7f11600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f11600744b0 0x7f11600769c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1164034000 tx=0x7f116402e000 comp rx=0 tx=0).stop 2024-04-25T19:12:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11740d5140 msgr2=0x7f11741612b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11740d5140 0x7f11741612b0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f117408eb10 tx=0x7f116800c820 comp rx=0 tx=0).stop 2024-04-25T19:12:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 shutdown_connections 2024-04-25T19:12:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f11600744b0 0x7f11600769c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f11741647c0 0x7f117415afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f11740d5140 0x7f11741612b0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 --2- 172.21.15.28:0/4151654865 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11740b7580 0x7f1174160d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.540+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 >> 172.21.15.28:0/4151654865 conn(0x7f11740d2760 msgr2=0x7f11740d5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:49.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.541+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 shutdown_connections 2024-04-25T19:12:49.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.541+0000 7f117a63c640 1 -- 172.21.15.28:0/4151654865 wait complete. 2024-04-25T19:12:49.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.664+0000 7fd6493f3640 1 Processor -- start 2024-04-25T19:12:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.664+0000 7fd6493f3640 1 -- start start 2024-04-25T19:12:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd6493f3640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6440d4910 0x7fd644161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd6493f3640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd644163e80 con 0x7fd6440b5130 2024-04-25T19:12:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd6493f3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd644164160 con 0x7fd64406e5e0 2024-04-25T19:12:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd6493f3640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fd644163c30 con 0x7fd6440d4910 2024-04-25T19:12:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd642ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6440b5130 0x7fd64406ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39766/0 (socket says 172.21.15.28:39766) 2024-04-25T19:12:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd642ffd640 1 -- 172.21.15.28:0/2121238597 learned_addr learned my addr 172.21.15.28:0/2121238597 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101588817 0 0) 0x7fd644163e80 con 0x7fd6440b5130 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd630003520 con 0x7fd6440b5130 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd6437fe640 1 --2- 172.21.15.28:0/2121238597 >> v2:172.21.15.53:3300/0 conn(0x7fd6440d4910 0x7fd644161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4102242680 0 0) 0x7fd630003520 con 0x7fd6440b5130 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 >> v2:172.21.15.53:3300/0 conn(0x7fd6440d4910 msgr2=0x7fd644161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 --2- 172.21.15.28:0/2121238597 >> v2:172.21.15.53:3300/0 conn(0x7fd6440d4910 0x7fd644161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 >> v1:172.21.15.53:6789/0 conn(0x7fd64406e5e0 legacy=0x7fd64415f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd644164440 con 0x7fd6440b5130 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fd63c001670 con 0x7fd6440b5130 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 >> v1:172.21.15.28:6789/0 conn(0x7fd6440b5130 legacy=0x7fd64406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6300039b0 0x7fd630003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd630004800 0x7fd630004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6300051c0 0x7fd63000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd644163c30 con 0x7fd6300051c0 2024-04-25T19:12:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd63000c160 con 0x7fd630004800 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.666+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd63000c360 con 0x7fd6300039b0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.667+0000 7fd6437fe640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6300051c0 0x7fd63000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.667+0000 7fd642ffd640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6300039b0 0x7fd630003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.667+0000 7fd6427fc640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd630004800 0x7fd630004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.667+0000 7fd6437fe640 1 -- 172.21.15.28:0/2121238597 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6300039b0 msgr2=0x7fd630003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.667+0000 7fd6437fe640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6300039b0 0x7fd630003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.667+0000 7fd6437fe640 1 -- 172.21.15.28:0/2121238597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd630004800 msgr2=0x7fd630004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.667+0000 7fd6437fe640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd630004800 0x7fd630004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.667+0000 7fd6437fe640 1 -- 172.21.15.28:0/2121238597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd630003520 con 0x7fd6300051c0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd6437fe640 1 -- 172.21.15.28:0/2121238597 reap_dead start 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd6437fe640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6300051c0 0x7fd63000b9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd634009fc0 tx=0x7fd630005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a2f76f53bafe74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634009c10 con 0x7fd6300051c0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd634003b90 con 0x7fd6300051c0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd641ffb640 1 -- 172.21.15.28:0/2121238597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634004020 con 0x7fd6300051c0 2024-04-25T19:12:49.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd6493f3640 1 -- 172.21.15.28:0/2121238597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6300051c0 msgr2=0x7fd63000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd6493f3640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6300051c0 0x7fd63000b9c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd634009fc0 tx=0x7fd630005830 comp rx=0 tx=0).stop 2024-04-25T19:12:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd6493f3640 1 -- 172.21.15.28:0/2121238597 shutdown_connections 2024-04-25T19:12:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd6493f3640 1 --2- 172.21.15.28:0/2121238597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6300051c0 0x7fd63000b9c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.668+0000 7fd6493f3640 1 -- 172.21.15.28:0/2121238597 >> 172.21.15.28:0/2121238597 conn(0x7fd6440b7ad0 msgr2=0x7fd6440b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.669+0000 7fd6493f3640 1 -- 172.21.15.28:0/2121238597 shutdown_connections 2024-04-25T19:12:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.669+0000 7fd6493f3640 1 -- 172.21.15.28:0/2121238597 wait complete. 2024-04-25T19:12:49.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.670+0000 7fd6493f3640 1 Processor -- start 2024-04-25T19:12:49.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.671+0000 7fd6493f3640 1 -- start start 2024-04-25T19:12:49.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6493f3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6300051c0 0x7fd644153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6493f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd644158db0 0x7fd64415c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6493f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64415ceb0 0x7fd6441572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6493f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd644161130 con 0x7fd644158db0 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6493f3640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd644164a90 con 0x7fd6300051c0 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6493f3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd6441612b0 con 0x7fd64415ceb0 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd644158db0 0x7fd64415c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd644158db0 0x7fd64415c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54142/0 (socket says 172.21.15.28:54142) 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 -- 172.21.15.28:0/487622026 learned_addr learned my addr 172.21.15.28:0/487622026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 -- 172.21.15.28:0/487622026 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64415ceb0 msgr2=0x7fd6441572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64415ceb0 0x7fd6441572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 -- 172.21.15.28:0/487622026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6300051c0 msgr2=0x7fd644153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd642ffd640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6300051c0 0x7fd644153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6300051c0 0x7fd644153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 -- 172.21.15.28:0/487622026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64415d2c0 con 0x7fd644158db0 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd6427fc640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd644158db0 0x7fd64415c970 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00f620 tx=0x7fd62c00fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.672+0000 7fd63b7fe640 1 -- 172.21.15.28:0/487622026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c0052e0 con 0x7fd644158db0 2024-04-25T19:12:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.673+0000 7fd63b7fe640 1 -- 172.21.15.28:0/487622026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd62c018070 con 0x7fd644158db0 2024-04-25T19:12:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.673+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd644154e20 con 0x7fd644158db0 2024-04-25T19:12:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.673+0000 7fd63b7fe640 1 -- 172.21.15.28:0/487622026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c003810 con 0x7fd644158db0 2024-04-25T19:12:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.673+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6441544d0 con 0x7fd644158db0 2024-04-25T19:12:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.674+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd644155090 con 0x7fd644158db0 2024-04-25T19:12:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.674+0000 7fd63b7fe640 1 -- 172.21.15.28:0/487622026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd62c0039b0 con 0x7fd644158db0 2024-04-25T19:12:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.675+0000 7fd63b7fe640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6240744e0 0x7fd6240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.675+0000 7fd63b7fe640 1 -- 172.21.15.28:0/487622026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd62c095130 con 0x7fd644158db0 2024-04-25T19:12:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.675+0000 7fd642ffd640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6240744e0 0x7fd6240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.675+0000 7fd642ffd640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6240744e0 0x7fd6240769f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd63c001050 tx=0x7fd63c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:49.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.680+0000 7fd63b7fe640 1 -- 172.21.15.28:0/487622026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd64415d2c0 con 0x7fd644158db0 2024-04-25T19:12:49.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.907+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd644050740 con 0x7fd644158db0 2024-04-25T19:12:49.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.907+0000 7fd63b7fe640 1 -- 172.21.15.28:0/487622026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5169 (secure 0 0 0) 0x7fd62c05f7c0 con 0x7fd644158db0 2024-04-25T19:12:49.909 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:49.910 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/2695669357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2695669357},{"type":"v1","addr":"172.21.15.53:6834","nonce":2695669357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/2737682189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2737682189},{"type":"v1","addr":"172.21.15.28:6836","nonce":2737682189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/6663801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":6663801},{"type":"v1","addr":"172.21.15.28:6837","nonce":6663801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:47.076286+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/803470605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":803470605},{"type":"v1","addr":"172.21.15.53:6835","nonce":803470605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":2}]} 2024-04-25T19:12:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.909+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6240744e0 msgr2=0x7fd6240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.909+0000 7fd6493f3640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6240744e0 0x7fd6240769f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd63c001050 tx=0x7fd63c009040 comp rx=0 tx=0).stop 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd644158db0 msgr2=0x7fd64415c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd644158db0 0x7fd64415c970 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00f620 tx=0x7fd62c00fae0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 shutdown_connections 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6240744e0 0x7fd6240769f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd64415ceb0 0x7fd6441572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd644158db0 0x7fd64415c970 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 --2- 172.21.15.28:0/487622026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6300051c0 0x7fd644153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 >> 172.21.15.28:0/487622026 conn(0x7fd6440b7ad0 msgr2=0x7fd644161d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:49.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 shutdown_connections 2024-04-25T19:12:49.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:49.910+0000 7fd6493f3640 1 -- 172.21.15.28:0/487622026 wait complete. 2024-04-25T19:12:50.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T19:12:44.063277+0000', 'modified': '2024-04-25T19:12:47.076286+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5325}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5325': {'gid': 5325, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6835/803470605', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6833', 'nonce': 803470605}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 803470605}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5325, 'qdb_cluster': [5325]} 2024-04-25T19:12:50.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T19:12:50.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T19:12:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.443+0000 7f31635b2640 1 Processor -- start 2024-04-25T19:12:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.443+0000 7f31635b2640 1 -- start start 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.444+0000 7f31635b2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f315c163c80 0x7f315c0d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.444+0000 7f31635b2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 0x7f315c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.444+0000 7f31635b2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f315c164090 con 0x7f315c160180 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.444+0000 7f31635b2640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f315c0d0c00 con 0x7f315c163c80 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.444+0000 7f31635b2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f315c0d4510 con 0x7f315c0d2550 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.444+0000 7f3160b4d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f315c163c80 0x7f315c0d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.444+0000 7f3160b4d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f315c163c80 0x7f315c0d06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54148/0 (socket says 172.21.15.28:54148) 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.444+0000 7f3160b4d640 1 -- 172.21.15.28:0/711201662 learned_addr learned my addr 172.21.15.28:0/711201662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.445+0000 7f3161b4f640 1 --2- 172.21.15.28:0/711201662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 0x7f315c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.445+0000 7f3160b4d640 1 -- 172.21.15.28:0/711201662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 msgr2=0x7f315c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.445+0000 7f3160b4d640 1 --2- 172.21.15.28:0/711201662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 0x7f315c0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.445+0000 7f3160b4d640 1 -- 172.21.15.28:0/711201662 >> v1:172.21.15.53:6789/0 conn(0x7f315c160180 legacy=0x7f315c0b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.445+0000 7f3160b4d640 1 -- 172.21.15.28:0/711201662 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c164bb0 con 0x7f315c163c80 2024-04-25T19:12:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.445+0000 7f3160b4d640 1 --2- 172.21.15.28:0/711201662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c0d06c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f314c009870 tx=0x7f314c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a17e8cb2712f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f3153fff640 1 -- 172.21.15.28:0/711201662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c037070 con 0x7f315c163c80 2024-04-25T19:12:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f3153fff640 1 -- 172.21.15.28:0/711201662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f314c032440 con 0x7f315c163c80 2024-04-25T19:12:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f3153fff640 1 -- 172.21.15.28:0/711201662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c032810 con 0x7f315c163c80 2024-04-25T19:12:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f31635b2640 1 -- 172.21.15.28:0/711201662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 msgr2=0x7f315c0d06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f31635b2640 1 --2- 172.21.15.28:0/711201662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c0d06c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f314c009870 tx=0x7f314c02a210 comp rx=0 tx=0).stop 2024-04-25T19:12:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f31635b2640 1 -- 172.21.15.28:0/711201662 shutdown_connections 2024-04-25T19:12:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f31635b2640 1 --2- 172.21.15.28:0/711201662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 0x7f315c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f31635b2640 1 --2- 172.21.15.28:0/711201662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c0d06c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.446+0000 7f31635b2640 1 -- 172.21.15.28:0/711201662 >> 172.21.15.28:0/711201662 conn(0x7f315c06c750 msgr2=0x7f315c06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.447+0000 7f31635b2640 1 -- 172.21.15.28:0/711201662 shutdown_connections 2024-04-25T19:12:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.447+0000 7f31635b2640 1 -- 172.21.15.28:0/711201662 wait complete. 2024-04-25T19:12:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.448+0000 7f31635b2640 1 Processor -- start 2024-04-25T19:12:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.448+0000 7f31635b2640 1 -- start start 2024-04-25T19:12:50.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f31635b2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 0x7f315c1f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f31635b2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f315c160180 0x7f315c1f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f31635b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c15cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f31635b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f315c1645f0 con 0x7f315c163c80 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f31635b2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f315c164310 con 0x7f315c160180 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f31635b2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f315c1648d0 con 0x7f315c0d2550 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c15cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c15cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54154/0 (socket says 172.21.15.28:54154) 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 -- 172.21.15.28:0/345111845 learned_addr learned my addr 172.21.15.28:0/345111845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3160b4d640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f315c160180 0x7f315c1f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f316134e640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 0x7f315c1f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 -- 172.21.15.28:0/345111845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 msgr2=0x7f315c1f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 0x7f315c1f6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 -- 172.21.15.28:0/345111845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f315c160180 msgr2=0x7f315c1f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f315c160180 0x7f315c1f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 -- 172.21.15.28:0/345111845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c0cc220 con 0x7f315c163c80 2024-04-25T19:12:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3161b4f640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c15cc90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3154009fd0 tx=0x7f315400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3151ffb640 1 -- 172.21.15.28:0/345111845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154014420 con 0x7f315c163c80 2024-04-25T19:12:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3151ffb640 1 -- 172.21.15.28:0/345111845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3154019070 con 0x7f315c163c80 2024-04-25T19:12:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f31635b2640 1 -- 172.21.15.28:0/345111845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315c0cc500 con 0x7f315c163c80 2024-04-25T19:12:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.450+0000 7f3151ffb640 1 -- 172.21.15.28:0/345111845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154014800 con 0x7f315c163c80 2024-04-25T19:12:50.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.451+0000 7f31635b2640 1 -- 172.21.15.28:0/345111845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c0ccac0 con 0x7f315c163c80 2024-04-25T19:12:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.453+0000 7f3151ffb640 1 -- 172.21.15.28:0/345111845 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f31540149a0 con 0x7f315c163c80 2024-04-25T19:12:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.453+0000 7f3151ffb640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31400744b0 0x7f31400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.453+0000 7f3151ffb640 1 -- 172.21.15.28:0/345111845 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f3154095a40 con 0x7f315c163c80 2024-04-25T19:12:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.453+0000 7f316134e640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31400744b0 0x7f31400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.454+0000 7f316134e640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31400744b0 0x7f31400769c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3158006fd0 tx=0x7f3158009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.454+0000 7f316a465740 1 Processor -- start 2024-04-25T19:12:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.454+0000 7f316a465740 1 -- start start 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f316a465740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 0x55c6e8c0c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f316a465740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c08fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f316a465740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8be2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f316a465740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c6e8b46740 con 0x55c6e881f590 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f316a465740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c6e8ae23a0 con 0x55c6e88173b0 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f316a465740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55c6e8b42700 con 0x55c6e8840270 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8be2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8be2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54156/0 (socket says 172.21.15.28:54156) 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 -- 172.21.15.28:0/2204523706 learned_addr learned my addr 172.21.15.28:0/2204523706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3160b4d640 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c08fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f316134e640 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 0x55c6e8c0c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 -- 172.21.15.28:0/2204523706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 msgr2=0x55c6e8c0c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 0x55c6e8c0c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 -- 172.21.15.28:0/2204523706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 msgr2=0x55c6e8c08fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c08fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 -- 172.21.15.28:0/2204523706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6e8afe750 con 0x55c6e881f590 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3161b4f640 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8be2260 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f315400c040 tx=0x7f315405f320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d9c31213c555ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.456+0000 7f3146ffd640 1 -- 172.21.15.28:0/2204523706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31540680f0 con 0x55c6e881f590 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f3146ffd640 1 -- 172.21.15.28:0/2204523706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31540687a0 con 0x55c6e881f590 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f3146ffd640 1 -- 172.21.15.28:0/2204523706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154068530 con 0x55c6e881f590 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 -- 172.21.15.28:0/2204523706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 msgr2=0x55c6e8be2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8be2260 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f315400c040 tx=0x7f315405f320 comp rx=0 tx=0).stop 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 -- 172.21.15.28:0/2204523706 shutdown_connections 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8be2260 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c08fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 --2- 172.21.15.28:0/2204523706 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 0x55c6e8c0c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 -- 172.21.15.28:0/2204523706 >> 172.21.15.28:0/2204523706 conn(0x55c6e88343c0 msgr2=0x55c6e8be19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 -- 172.21.15.28:0/2204523706 shutdown_connections 2024-04-25T19:12:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.457+0000 7f316a465740 1 -- 172.21.15.28:0/2204523706 wait complete. 2024-04-25T19:12:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.458+0000 7f316a465740 1 -- start start 2024-04-25T19:12:50.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.458+0000 7f316a465740 1 Processor -- start 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f316a465740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f316a465740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 0x55c6e8c24cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f316a465740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c0f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f316a465740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c6e8acdfe0 con 0x55c6e8840270 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f316a465740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c6e8c05a10 con 0x55c6e88173b0 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f316a465740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55c6e8c05c60 con 0x55c6e881f590 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f316a465740 10 client.0 _ll_register_callbacks cb 0x55c6e8ae4150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c0f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c0f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54170/0 (socket says 172.21.15.28:54170) 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 -- 172.21.15.28:0/1867197196 learned_addr learned my addr 172.21.15.28:0/1867197196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f316134e640 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3160b4d640 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 0x55c6e8c24cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 -- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 msgr2=0x55c6e8c24cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 0x55c6e8c24cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 -- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 msgr2=0x55c6e8c053b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c053b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 -- 172.21.15.28:0/1867197196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6e88582d0 con 0x55c6e8840270 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f3161b4f640 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c0f830 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f31540094c0 tx=0x7f3154007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9d476d7303e10b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.460+0000 7f313bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f313bfff640 1 -- 172.21.15.28:0/1867197196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154068520 con 0x55c6e8840270 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f313bfff640 1 -- 172.21.15.28:0/1867197196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f315405ece0 con 0x55c6e8840270 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 1 -- 172.21.15.28:0/1867197196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31540687a0 con 0x55c6e8840270 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f313bfff640 1 -- 172.21.15.28:0/1867197196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f3154081080 con 0x55c6e8840270 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 10 client.5502 fetch_fsmap learned FSMap version 18 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 1 -- 172.21.15.28:0/1867197196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x55c6e89226b0 con 0x55c6e8840270 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f313bfff640 1 -- 172.21.15.28:0/1867197196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31540687a0 con 0x55c6e8840270 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f313bfff640 1 -- 172.21.15.28:0/1867197196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7f315405b3f0 con 0x55c6e8840270 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 10 client.5502 fetch_fsmap finished waiting for FSMap version 18 2024-04-25T19:12:50.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 10 client.5502 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 20 client.5502 populate_metadata read hostname 'smithi028' 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] conn(0x55c6e87fee40 0x55c6e8c39880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 4 client.5502 mds_command: new command op to 5325 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316a465740 1 -- 172.21.15.28:0/1867197196 --> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6e8b54350 con 0x55c6e87fee40 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.461+0000 7f316134e640 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] conn(0x55c6e87fee40 0x55c6e8c39880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.462+0000 7f313bfff640 10 client.5502 ms_handle_connect on v2:172.21.15.53:6833/803470605 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.462+0000 7f316134e640 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] conn(0x55c6e87fee40 0x55c6e8c39880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d75003e02f38471 server_cookie=3ad37ab0b8eac9ce in_seq=0 out_seq=0 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f313bfff640 1 -- 172.21.15.28:0/1867197196 <== mds.0 v2:172.21.15.53:6833/803470605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c6e8b54350 con 0x55c6e87fee40 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f313bfff640 10 client.5502 handle_command_reply: tid=0 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 client.5502 shutdown 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 -- 172.21.15.28:0/1867197196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 msgr2=0x55c6e8c0f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c0f830 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f31540094c0 tx=0x7f3154007f60 comp rx=0 tx=0).stop 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 -- 172.21.15.28:0/1867197196 shutdown_connections 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] conn(0x55c6e87fee40 0x55c6e8c39880 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c0f830 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 0x55c6e8c24cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f313bfff640 0 client.5502 ms_handle_reset on v2:172.21.15.53:6833/803470605 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 --2- 172.21.15.28:0/1867197196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 -- 172.21.15.28:0/1867197196 >> 172.21.15.28:0/1867197196 conn(0x55c6e88343c0 msgr2=0x55c6e8be19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 -- 172.21.15.28:0/1867197196 shutdown_connections 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 1 -- 172.21.15.28:0/1867197196 wait complete. 2024-04-25T19:12:50.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.463+0000 7f316a465740 20 client.5502 trim_cache size 0 max 16384 2024-04-25T19:12:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.482+0000 7f316a465740 1 Processor -- start 2024-04-25T19:12:50.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.482+0000 7f316a465740 1 -- start start 2024-04-25T19:12:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f316a465740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c00a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f316a465740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8c2a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f316a465740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 0x55c6e8c2aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f316a465740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f315c202510 con 0x55c6e881f590 2024-04-25T19:12:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f316a465740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f315c0cade0 con 0x55c6e88173b0 2024-04-25T19:12:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f316a465740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f315c0cd9f0 con 0x55c6e8840270 2024-04-25T19:12:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8c2a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8c2a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54172/0 (socket says 172.21.15.28:54172) 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 -- 172.21.15.28:0/803227524 learned_addr learned my addr 172.21.15.28:0/803227524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f316134e640 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c00a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3161b4f640 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 0x55c6e8c2aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 -- 172.21.15.28:0/803227524 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 msgr2=0x55c6e8c2aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 0x55c6e8c2aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 -- 172.21.15.28:0/803227524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 msgr2=0x55c6e8c00a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c00a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 -- 172.21.15.28:0/803227524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6e89226b0 con 0x55c6e881f590 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f3160b4d640 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8c2a520 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f314c009990 tx=0x7f314c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2014a79a57bce4b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f31467fc640 1 -- 172.21.15.28:0/803227524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c037040 con 0x55c6e881f590 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f31467fc640 1 -- 172.21.15.28:0/803227524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f314c0328d0 con 0x55c6e881f590 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.484+0000 7f31467fc640 1 -- 172.21.15.28:0/803227524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f314c032cc0 con 0x55c6e881f590 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 -- 172.21.15.28:0/803227524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 msgr2=0x55c6e8c2a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8c2a520 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f314c009990 tx=0x7f314c02c000 comp rx=0 tx=0).stop 2024-04-25T19:12:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 -- 172.21.15.28:0/803227524 shutdown_connections 2024-04-25T19:12:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e8840270 0x55c6e8c2aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e881f590 0x55c6e8c2a520 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 --2- 172.21.15.28:0/803227524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c00a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 -- 172.21.15.28:0/803227524 >> 172.21.15.28:0/803227524 conn(0x55c6e88343c0 msgr2=0x55c6e8be00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 -- 172.21.15.28:0/803227524 shutdown_connections 2024-04-25T19:12:50.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.485+0000 7f316a465740 1 -- 172.21.15.28:0/803227524 wait complete. 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.486+0000 7f316a465740 1 -- start start 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.486+0000 7f316a465740 1 Processor -- start 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.487+0000 7f316a465740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c24180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f316a465740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 0x55c6e8ca3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f316a465740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c45d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f316a465740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c6e8c10100 con 0x55c6e8840270 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f316a465740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c6e8acdfe0 con 0x55c6e88173b0 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f316a465740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55c6e8c0b2b0 con 0x55c6e881f590 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f316a465740 10 client.0 _ll_register_callbacks cb 0x7f315405f1f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c45d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c45d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54180/0 (socket says 172.21.15.28:54180) 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 -- 172.21.15.28:0/670731003 learned_addr learned my addr 172.21.15.28:0/670731003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:50.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3160b4d640 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 0x55c6e8ca3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f316134e640 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c24180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 -- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 msgr2=0x55c6e8ca3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 0x55c6e8ca3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 -- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 msgr2=0x55c6e8c24180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c24180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 -- 172.21.15.28:0/670731003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6e8c0d9d0 con 0x55c6e8840270 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f3161b4f640 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c45d20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3154081050 tx=0x7f3154020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31ab63bb0fc88ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f313bfff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f313bfff640 1 -- 172.21.15.28:0/670731003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315405c0d0 con 0x55c6e8840270 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f313bfff640 1 -- 172.21.15.28:0/670731003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f315405eef0 con 0x55c6e8840270 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f316a465740 1 -- 172.21.15.28:0/670731003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f314c0328d0 con 0x55c6e8840270 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.488+0000 7f313bfff640 1 -- 172.21.15.28:0/670731003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f31540680f0 con 0x55c6e8840270 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316a465740 10 client.5508 fetch_fsmap learned FSMap version 18 2024-04-25T19:12:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316a465740 1 -- 172.21.15.28:0/670731003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) -- 0x55c6e8c0d9d0 con 0x55c6e8840270 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f313bfff640 1 -- 172.21.15.28:0/670731003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f314c0328d0 con 0x55c6e8840270 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f313bfff640 1 -- 172.21.15.28:0/670731003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 18) ==== 3494+0+0 (secure 0 0 0) 0x7f315405e750 con 0x55c6e8840270 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316a465740 10 client.5508 fetch_fsmap finished waiting for FSMap version 18 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316a465740 10 client.5508 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316a465740 20 client.5508 populate_metadata read hostname 'smithi028' 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316a465740 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] conn(0x55c6e8c2a930 0x55c6e8c48b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316a465740 4 client.5508 mds_command: new command op to 5325 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316a465740 1 -- 172.21.15.28:0/670731003 --> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] -- command(tid 0: {"prefix": "status"}) -- 0x55c6e8b4e1c0 con 0x55c6e8c2a930 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316134e640 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] conn(0x55c6e8c2a930 0x55c6e8c48b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f313bfff640 10 client.5508 ms_handle_connect on v2:172.21.15.53:6833/803470605 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.489+0000 7f316134e640 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] conn(0x55c6e8c2a930 0x55c6e8c48b80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d1993ffe0232447 server_cookie=15639eb0a900881c in_seq=0 out_seq=0 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f313bfff640 1 -- 172.21.15.28:0/670731003 <== mds.0 v2:172.21.15.53:6833/803470605 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55c6e8b4e1c0 con 0x55c6e8c2a930 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f313bfff640 10 client.5508 handle_command_reply: tid=0 2024-04-25T19:12:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 client.5508 shutdown 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 -- 172.21.15.28:0/670731003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 msgr2=0x55c6e8c45d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c45d20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3154081050 tx=0x7f3154020000 comp rx=0 tx=0).stop 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 -- 172.21.15.28:0/670731003 shutdown_connections 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:6833/803470605,v1:172.21.15.53:6835/803470605] conn(0x55c6e8c2a930 0x55c6e8c48b80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c6e8840270 0x55c6e8c45d20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c6e881f590 0x55c6e8ca3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f313bfff640 0 client.5508 ms_handle_reset on v2:172.21.15.53:6833/803470605 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 --2- 172.21.15.28:0/670731003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c6e88173b0 0x55c6e8c24180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 -- 172.21.15.28:0/670731003 >> 172.21.15.28:0/670731003 conn(0x55c6e88343c0 msgr2=0x55c6e8c0bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 -- 172.21.15.28:0/670731003 shutdown_connections 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 1 -- 172.21.15.28:0/670731003 wait complete. 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.490+0000 7f316a465740 20 client.5508 trim_cache size 0 max 16384 2024-04-25T19:12:50.496 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "5d8007d4-24a5-41be-aa4d-cd64fedac62a", 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5325, 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 5.8215639819999998, 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 18, 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 38, 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 38, 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 13.685272445000001 2024-04-25T19:12:50.497 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 -- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31400744b0 msgr2=0x7f31400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31400744b0 0x7f31400769c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3158006fd0 tx=0x7f3158009060 comp rx=0 tx=0).stop 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 -- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 msgr2=0x7f315c15cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c15cc90 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3154009fd0 tx=0x7f315400e4b0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 -- 172.21.15.28:0/345111845 shutdown_connections 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31400744b0 0x7f31400769c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f315c163c80 0x7f315c15cc90 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f315c160180 0x7f315c1f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 --2- 172.21.15.28:0/345111845 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f315c0d2550 0x7f315c1f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 -- 172.21.15.28:0/345111845 >> 172.21.15.28:0/345111845 conn(0x7f315c06c750 msgr2=0x7f315c0b72d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 -- 172.21.15.28:0/345111845 shutdown_connections 2024-04-25T19:12:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.491+0000 7f31635b2640 1 -- 172.21.15.28:0/345111845 wait complete. 2024-04-25T19:12:50.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T19:12:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.614+0000 7feda07ce640 1 Processor -- start 2024-04-25T19:12:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.614+0000 7feda07ce640 1 -- start start 2024-04-25T19:12:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.615+0000 7feda07ce640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fed9815f180 0x7fed98161c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7feda07ce640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed981621a0 0x7fed9806d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7feda07ce640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed9806ded0 con 0x7fed9806dac0 2024-04-25T19:12:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7feda07ce640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fed9815f590 con 0x7fed9815f180 2024-04-25T19:12:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7feda07ce640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fed98158ae0 con 0x7fed981621a0 2024-04-25T19:12:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9dd69640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed981621a0 0x7fed9806d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9e56a640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fed9815f180 0x7fed98161c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9ed6b640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fed9806dac0 0x7fed9806f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:41580/0 (socket says 172.21.15.28:41580) 2024-04-25T19:12:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9dd69640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed981621a0 0x7fed9806d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:55686/0 (socket says 172.21.15.28:55686) 2024-04-25T19:12:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9ed6b640 1 -- 172.21.15.28:0/1800639435 learned_addr learned my addr 172.21.15.28:0/1800639435 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9d568640 1 -- 172.21.15.28:0/1800639435 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2133071518 0 0) 0x7fed88001490 con 0x7fed9806dac0 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9d568640 1 -- 172.21.15.28:0/1800639435 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed80003520 con 0x7fed9806dac0 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9dd69640 1 -- 172.21.15.28:0/1800639435 >> v2:172.21.15.53:3300/0 conn(0x7fed9815f180 msgr2=0x7fed98161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9dd69640 1 --2- 172.21.15.28:0/1800639435 >> v2:172.21.15.53:3300/0 conn(0x7fed9815f180 0x7fed98161c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9dd69640 1 -- 172.21.15.28:0/1800639435 >> v1:172.21.15.53:6789/0 conn(0x7fed9806dac0 legacy=0x7fed9806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.616+0000 7fed9dd69640 1 -- 172.21.15.28:0/1800639435 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed98162700 con 0x7fed981621a0 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.617+0000 7fed9dd69640 1 --2- 172.21.15.28:0/1800639435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed981621a0 0x7fed9806d580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0097a0 tx=0x7fed8c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=102a3bd248d25495 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.617+0000 7fed9d568640 1 -- 172.21.15.28:0/1800639435 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c037070 con 0x7fed981621a0 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.617+0000 7fed9d568640 1 -- 172.21.15.28:0/1800639435 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed8c032440 con 0x7fed981621a0 2024-04-25T19:12:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.617+0000 7fed9d568640 1 -- 172.21.15.28:0/1800639435 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed8c0327f0 con 0x7fed981621a0 2024-04-25T19:12:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.618+0000 7feda07ce640 1 -- 172.21.15.28:0/1800639435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed981621a0 msgr2=0x7fed9806d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.618+0000 7feda07ce640 1 --2- 172.21.15.28:0/1800639435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed981621a0 0x7fed9806d580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fed8c0097a0 tx=0x7fed8c02a180 comp rx=0 tx=0).stop 2024-04-25T19:12:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.618+0000 7feda07ce640 1 -- 172.21.15.28:0/1800639435 shutdown_connections 2024-04-25T19:12:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.618+0000 7feda07ce640 1 --2- 172.21.15.28:0/1800639435 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed981621a0 0x7fed9806d580 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.618+0000 7feda07ce640 1 --2- 172.21.15.28:0/1800639435 >> v2:172.21.15.53:3300/0 conn(0x7fed9815f180 0x7fed98161c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.618+0000 7feda07ce640 1 -- 172.21.15.28:0/1800639435 >> 172.21.15.28:0/1800639435 conn(0x7fed980732b0 msgr2=0x7fed980736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.618+0000 7feda07ce640 1 -- 172.21.15.28:0/1800639435 shutdown_connections 2024-04-25T19:12:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.618+0000 7feda07ce640 1 -- 172.21.15.28:0/1800639435 wait complete. 2024-04-25T19:12:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.620+0000 7feda07ce640 1 Processor -- start 2024-04-25T19:12:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.620+0000 7feda07ce640 1 -- start start 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7feda07ce640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed9806dac0 0x7fed9813ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7feda07ce640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed9815f180 0x7fed9813bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7feda07ce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed981621a0 0x7fed9806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7feda07ce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed9815ec60 con 0x7fed981621a0 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7feda07ce640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fed9815e980 con 0x7fed9815f180 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7feda07ce640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fed9815e670 con 0x7fed9806dac0 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7fed9ed6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed981621a0 0x7fed9806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7fed9ed6b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed981621a0 0x7fed9806cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54186/0 (socket says 172.21.15.28:54186) 2024-04-25T19:12:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7fed9ed6b640 1 -- 172.21.15.28:0/3569613857 learned_addr learned my addr 172.21.15.28:0/3569613857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7fed9dd69640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed9815f180 0x7fed9813bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.621+0000 7fed9e56a640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed9806dac0 0x7fed9813ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed9ed6b640 1 -- 172.21.15.28:0/3569613857 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed9806dac0 msgr2=0x7fed9813ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed9ed6b640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed9806dac0 0x7fed9813ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed9ed6b640 1 -- 172.21.15.28:0/3569613857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed9815f180 msgr2=0x7fed9813bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed9ed6b640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed9815f180 0x7fed9813bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed9ed6b640 1 -- 172.21.15.28:0/3569613857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed9813f4c0 con 0x7fed981621a0 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed9ed6b640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed981621a0 0x7fed9806cb30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed8800ca70 tx=0x7fed8800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed96ffd640 1 -- 172.21.15.28:0/3569613857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88017070 con 0x7fed981621a0 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed96ffd640 1 -- 172.21.15.28:0/3569613857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed8800abd0 con 0x7fed981621a0 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed981f4980 con 0x7fed981621a0 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7fed96ffd640 1 -- 172.21.15.28:0/3569613857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed88012400 con 0x7fed981621a0 2024-04-25T19:12:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.622+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed981f4f40 con 0x7fed981621a0 2024-04-25T19:12:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.623+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed981f46f0 con 0x7fed981621a0 2024-04-25T19:12:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.624+0000 7fed96ffd640 1 -- 172.21.15.28:0/3569613857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fed8801a070 con 0x7fed981621a0 2024-04-25T19:12:50.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.624+0000 7fed96ffd640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fed840744b0 0x7fed840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.624+0000 7fed96ffd640 1 -- 172.21.15.28:0/3569613857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fed88095110 con 0x7fed981621a0 2024-04-25T19:12:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.624+0000 7fed9e56a640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fed840744b0 0x7fed840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:50.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.625+0000 7fed9e56a640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fed840744b0 0x7fed840769c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fed98070310 tx=0x7fed90009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.629+0000 7fed96ffd640 1 -- 172.21.15.28:0/3569613857 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed9813f4c0 con 0x7fed981621a0 2024-04-25T19:12:50.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.855+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fed98050740 con 0x7fed981621a0 2024-04-25T19:12:50.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.855+0000 7fed96ffd640 1 -- 172.21.15.28:0/3569613857 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) ==== 53+0+73 (secure 0 0 0) 0x7fed9806ded0 con 0x7fed981621a0 2024-04-25T19:12:50.857 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T19:12:50.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fed840744b0 msgr2=0x7fed840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fed840744b0 0x7fed840769c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fed98070310 tx=0x7fed90009190 comp rx=0 tx=0).stop 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed981621a0 msgr2=0x7fed9806cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed981621a0 0x7fed9806cb30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fed8800ca70 tx=0x7fed8800cf30 comp rx=0 tx=0).stop 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 shutdown_connections 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fed840744b0 0x7fed840769c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed981621a0 0x7fed9806cb30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed9815f180 0x7fed9813bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 --2- 172.21.15.28:0/3569613857 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fed9806dac0 0x7fed9813ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 >> 172.21.15.28:0/3569613857 conn(0x7fed980732b0 msgr2=0x7fed980b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 shutdown_connections 2024-04-25T19:12:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:50.857+0000 7feda07ce640 1 -- 172.21.15.28:0/3569613857 wait complete. 2024-04-25T19:12:50.870 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T19:12:50.871 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T19:12:50.871 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-25T19:12:50.871 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T19:12:50.871 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:50.871 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T19:12:50.871 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T19:12:50.871 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T19:12:50.871 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T19:12:50.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:50.871 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-25T19:12:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:12:50.890 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:12:50.890 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:12:50.890 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:50.890 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T19:12:50.890 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:50.890 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:50.890 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42689sec preferred_lft 42689sec 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-25T19:12:50.891 INFO:teuthology.orchestra.run.smithi028.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether b6:8b:9a:39:76:98 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::d015:3fff:fe07:1fb/64 scope link 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether b6:8b:9a:39:76:98 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::b48b:9aff:fe39:7698/64 scope link 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether e6:ae:c1:be:ae:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::e4ae:c1ff:febe:aec4/64 scope link 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:12:50.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:50.893 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T19:12:50.954 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-25T19:12:50.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:50.956 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-25T19:12:51.011 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-25T19:12:51.011 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T19:12:51.012 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T19:12:51.012 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T19:12:51.012 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T19:12:51.012 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:12:51.012 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:12:51.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T19:12:51.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T19:12:51.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T19:12:51.012 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:12:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:12:51.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:51.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:51.138 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:12:51.153 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T19:12:51.153 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T19:12:51.153 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T19:12:51.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:51.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-25T19:12:51.218 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:51.218 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:51.272 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T19:12:51.273 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:12:51.273 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:51.327 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T19:12:51.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:51.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:51.422 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:51.433 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:12:51.433 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:12:51.433 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.53:3300,172.21.15.53:3301" 2024-04-25T19:12:51.433 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@5d8007d4-24a5-41be-aa4d-cd64fedac62a.cephfs=/ 2024-04-25T19:12:51.433 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.53:3300/172.21.15.53:3301" will pass to kernel 2024-04-25T19:12:51.433 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T19:12:51.433 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-04-25T19:12:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T19:12:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T19:12:51.434 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T19:12:51.435 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T19:12:51.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:51.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:51.460 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:12:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:51 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:12:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:51.562 INFO:teuthology.orchestra.run.smithi028.stdout:client5302 2024-04-25T19:12:51.563 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:51.563 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:51.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:51.579 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-04-25T19:12:51.640 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:51.640 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:51.697 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:51.698 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=subdir/sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-04-25T19:12:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:6+0 records in 2024-04-25T19:12:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:6+0 records out 2024-04-25T19:12:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0391635 s, 161 MB/s 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> stdin-killer -- python3 -c ' 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> import json 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> import sys 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> except OSError as e: 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> sys.exit(e.errno) 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T19:12:51.795 DEBUG:teuthology.orchestra.run.smithi028:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-04-25T19:12:51.796 DEBUG:teuthology.orchestra.run.smithi028:> print(json.dumps( 2024-04-25T19:12:51.796 DEBUG:teuthology.orchestra.run.smithi028:> dict([(a, getattr(s, a)) for a in attrs]), 2024-04-25T19:12:51.796 DEBUG:teuthology.orchestra.run.smithi028:> indent=2)) 2024-04-25T19:12:51.796 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-04-25T19:12:51.864 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:12:51.864 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mode": 33188, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ino": 1099511627777, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_dev": 42, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_nlink": 1, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_uid": 1000, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_gid": 1316, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_size": 6291456, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_atime": 1714072371.7521315, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mtime": 1714072371.7711308, 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ctime": 1714072371.7711308 2024-04-25T19:12:51.865 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:12:51.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:12:51.873 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:51.873 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:51.930 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:12:51.930 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:12:51.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:12:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.135+0000 7fe164ce0640 1 Processor -- start 2024-04-25T19:12:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.135+0000 7fe164ce0640 1 -- start start 2024-04-25T19:12:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe164ce0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe16009bc50 0x7fe1600af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe164ce0640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe1601604b0 0x7fe1600b2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe164ce0640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1600d2f20 con 0x7fe16006ebd0 2024-04-25T19:12:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe164ce0640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fe1600b00b0 con 0x7fe16009bc50 2024-04-25T19:12:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe164ce0640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fe1600b0670 con 0x7fe1601604b0 2024-04-25T19:12:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe157fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe16009bc50 0x7fe1600af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe157fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe16009bc50 0x7fe1600af6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52260/0 (socket says 172.21.15.28:52260) 2024-04-25T19:12:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe157fff640 1 -- 172.21.15.28:0/952364828 learned_addr learned my addr 172.21.15.28:0/952364828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe15ed9d640 1 --2- 172.21.15.28:0/952364828 >> v2:172.21.15.53:3300/0 conn(0x7fe1601604b0 0x7fe1600b2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe157fff640 1 -- 172.21.15.28:0/952364828 >> v2:172.21.15.53:3300/0 conn(0x7fe1601604b0 msgr2=0x7fe1600b2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe157fff640 1 --2- 172.21.15.28:0/952364828 >> v2:172.21.15.53:3300/0 conn(0x7fe1601604b0 0x7fe1600b2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe157fff640 1 -- 172.21.15.28:0/952364828 >> v1:172.21.15.53:6789/0 conn(0x7fe16006ebd0 legacy=0x7fe16009b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:52.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.137+0000 7fe157fff640 1 -- 172.21.15.28:0/952364828 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1600d3200 con 0x7fe16009bc50 2024-04-25T19:12:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.138+0000 7fe157fff640 1 --2- 172.21.15.28:0/952364828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16009bc50 0x7fe1600af6a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe160074170 tx=0x7fe14802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca00de76d63437f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.138+0000 7fe15dd9b640 1 -- 172.21.15.28:0/952364828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe148037070 con 0x7fe16009bc50 2024-04-25T19:12:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.138+0000 7fe15dd9b640 1 -- 172.21.15.28:0/952364828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe148032440 con 0x7fe16009bc50 2024-04-25T19:12:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.138+0000 7fe15dd9b640 1 -- 172.21.15.28:0/952364828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe148032810 con 0x7fe16009bc50 2024-04-25T19:12:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.139+0000 7fe164ce0640 1 -- 172.21.15.28:0/952364828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16009bc50 msgr2=0x7fe1600af6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.139+0000 7fe164ce0640 1 --2- 172.21.15.28:0/952364828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16009bc50 0x7fe1600af6a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fe160074170 tx=0x7fe14802a230 comp rx=0 tx=0).stop 2024-04-25T19:12:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.139+0000 7fe164ce0640 1 -- 172.21.15.28:0/952364828 shutdown_connections 2024-04-25T19:12:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.139+0000 7fe164ce0640 1 --2- 172.21.15.28:0/952364828 >> v2:172.21.15.53:3300/0 conn(0x7fe1601604b0 0x7fe1600b2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.139+0000 7fe164ce0640 1 --2- 172.21.15.28:0/952364828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16009bc50 0x7fe1600af6a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.139+0000 7fe164ce0640 1 -- 172.21.15.28:0/952364828 >> 172.21.15.28:0/952364828 conn(0x7fe1600b7560 msgr2=0x7fe1600b7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.139+0000 7fe164ce0640 1 -- 172.21.15.28:0/952364828 shutdown_connections 2024-04-25T19:12:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.139+0000 7fe164ce0640 1 -- 172.21.15.28:0/952364828 wait complete. 2024-04-25T19:12:52.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.141+0000 7fe164ce0640 1 Processor -- start 2024-04-25T19:12:52.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.141+0000 7fe164ce0640 1 -- start start 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe164ce0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe16006ebd0 0x7fe16009df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe164ce0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1601604b0 0x7fe16006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe164ce0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16006e210 0x7fe1601479d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe164ce0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1600b3210 con 0x7fe16006e210 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe164ce0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe1600b0300 con 0x7fe1601604b0 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe164ce0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe1600b3390 con 0x7fe16006ebd0 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16006e210 0x7fe1601479d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16006e210 0x7fe1601479d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52264/0 (socket says 172.21.15.28:52264) 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15ed9d640 1 -- 172.21.15.28:0/2950809124 learned_addr learned my addr 172.21.15.28:0/2950809124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:52.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe157fff640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1601604b0 0x7fe16006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15e59c640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe16006ebd0 0x7fe16009df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15ed9d640 1 -- 172.21.15.28:0/2950809124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe16006ebd0 msgr2=0x7fe16009df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15ed9d640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe16006ebd0 0x7fe16009df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15ed9d640 1 -- 172.21.15.28:0/2950809124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1601604b0 msgr2=0x7fe16006dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15ed9d640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1601604b0 0x7fe16006dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.142+0000 7fe15ed9d640 1 -- 172.21.15.28:0/2950809124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe16006e620 con 0x7fe16006e210 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.143+0000 7fe15ed9d640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16006e210 0x7fe1601479d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe158007e60 tx=0x7fe15800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.143+0000 7fe156ffd640 1 -- 172.21.15.28:0/2950809124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158014420 con 0x7fe16006e210 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.143+0000 7fe156ffd640 1 -- 172.21.15.28:0/2950809124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe158019070 con 0x7fe16006e210 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.143+0000 7fe156ffd640 1 -- 172.21.15.28:0/2950809124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe158014820 con 0x7fe16006e210 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.143+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe16009cac0 con 0x7fe16006e210 2024-04-25T19:12:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.143+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe16009d080 con 0x7fe16006e210 2024-04-25T19:12:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.144+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe160050740 con 0x7fe16006e210 2024-04-25T19:12:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.145+0000 7fe156ffd640 1 -- 172.21.15.28:0/2950809124 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe1580149c0 con 0x7fe16006e210 2024-04-25T19:12:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.145+0000 7fe156ffd640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1400744b0 0x7fe1400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.145+0000 7fe156ffd640 1 -- 172.21.15.28:0/2950809124 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fe158095c50 con 0x7fe16006e210 2024-04-25T19:12:52.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.145+0000 7fe15e59c640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1400744b0 0x7fe1400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.146+0000 7fe15e59c640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1400744b0 0x7fe1400769c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe150006fd0 tx=0x7fe150008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.150+0000 7fe156ffd640 1 -- 172.21.15.28:0/2950809124 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe15809b020 con 0x7fe16006e210 2024-04-25T19:12:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.336+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1600b2550 con 0x7fe16006e210 2024-04-25T19:12:52.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.338+0000 7fe156ffd640 1 -- 172.21.15.28:0/2950809124 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16882 (secure 0 0 0) 0x7fe16006e620 con 0x7fe16006e210 2024-04-25T19:12:52.340 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:52.340 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":38,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:45.682011+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:12:52.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1400744b0 msgr2=0x7fe1400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1400744b0 0x7fe1400769c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe150006fd0 tx=0x7fe150008040 comp rx=0 tx=0).stop 2024-04-25T19:12:52.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16006e210 msgr2=0x7fe1601479d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16006e210 0x7fe1601479d0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe158007e60 tx=0x7fe15800e4b0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 shutdown_connections 2024-04-25T19:12:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1400744b0 0x7fe1400769c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe16006e210 0x7fe1601479d0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1601604b0 0x7fe16006dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 --2- 172.21.15.28:0/2950809124 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe16006ebd0 0x7fe16009df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 >> 172.21.15.28:0/2950809124 conn(0x7fe1600b7560 msgr2=0x7fe16006cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 shutdown_connections 2024-04-25T19:12:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.340+0000 7fe164ce0640 1 -- 172.21.15.28:0/2950809124 wait complete. 2024-04-25T19:12:52.353 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:12:41.809696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:12:43.014197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:12:52.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:12:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.467+0000 7f05ec834640 1 Processor -- start 2024-04-25T19:12:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.467+0000 7f05ec834640 1 -- start start 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05ec834640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f05e8160310 0x7f05e80d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05ec834640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05e8160720 con 0x7f05e80b79f0 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05ec834640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05e80d5640 con 0x7f05e80b9920 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05ec834640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f05e80d0f40 con 0x7f05e8160310 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e5d9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f05e80b79f0 0x7f05e80b7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38372/0 (socket says 172.21.15.28:38372) 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e5d9b640 1 -- 172.21.15.28:0/2351232108 learned_addr learned my addr 172.21.15.28:0/2351232108 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e659c640 1 --2- 172.21.15.28:0/2351232108 >> v2:172.21.15.53:3300/0 conn(0x7f05e8160310 0x7f05e80d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2317341477 0 0) 0x7f05e8160720 con 0x7f05e80b79f0 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05d0003520 con 0x7f05e80b79f0 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3457512440 0 0) 0x7f05d0003520 con 0x7f05e80b79f0 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 >> v2:172.21.15.53:3300/0 conn(0x7f05e8160310 msgr2=0x7f05e80d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 --2- 172.21.15.28:0/2351232108 >> v2:172.21.15.53:3300/0 conn(0x7f05e8160310 0x7f05e80d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 >> v1:172.21.15.53:6789/0 conn(0x7f05e80b9920 legacy=0x7f05e80b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e80d1220 con 0x7f05e80b79f0 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f05dc001670 con 0x7f05e80b79f0 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.469+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 >> v1:172.21.15.28:6789/0 conn(0x7f05e80b79f0 legacy=0x7f05e80b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e559a640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d00039b0 0x7f05d0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e559a640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0004800 0x7f05d0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e559a640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00051c0 0x7f05d000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05e80d0f40 con 0x7f05d0004800 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f05d000c160 con 0x7f05d00039b0 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f05d000c360 con 0x7f05d00051c0 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e5d9b640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0004800 0x7f05d0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e659c640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d00039b0 0x7f05d0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e6d9d640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00051c0 0x7f05d000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e5d9b640 1 -- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00051c0 msgr2=0x7f05d000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e5d9b640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00051c0 0x7f05d000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e5d9b640 1 -- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d00039b0 msgr2=0x7f05d0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e5d9b640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d00039b0 0x7f05d0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e5d9b640 1 -- 172.21.15.28:0/2351232108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d0003520 con 0x7f05d0004800 2024-04-25T19:12:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e6d9d640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05d00051c0 0x7f05d000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.470+0000 7f05e6d9d640 1 -- 172.21.15.28:0/2351232108 reap_dead start 2024-04-25T19:12:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.471+0000 7f05e5d9b640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0004800 0x7f05d0004c10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00c210 tx=0x7f05d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e30969a971d3a6d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.471+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc007d10 con 0x7f05d0004800 2024-04-25T19:12:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.471+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05dc0058d0 con 0x7f05d0004800 2024-04-25T19:12:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.471+0000 7f05e559a640 1 -- 172.21.15.28:0/2351232108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc005d40 con 0x7f05d0004800 2024-04-25T19:12:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.472+0000 7f05ec834640 1 -- 172.21.15.28:0/2351232108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0004800 msgr2=0x7f05d0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.472+0000 7f05ec834640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0004800 0x7f05d0004c10 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f05dc00c210 tx=0x7f05d0005830 comp rx=0 tx=0).stop 2024-04-25T19:12:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.472+0000 7f05ec834640 1 -- 172.21.15.28:0/2351232108 shutdown_connections 2024-04-25T19:12:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.472+0000 7f05ec834640 1 --2- 172.21.15.28:0/2351232108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0004800 0x7f05d0004c10 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.472+0000 7f05ec834640 1 -- 172.21.15.28:0/2351232108 >> 172.21.15.28:0/2351232108 conn(0x7f05e80d2a50 msgr2=0x7f05e80d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.472+0000 7f05ec834640 1 -- 172.21.15.28:0/2351232108 shutdown_connections 2024-04-25T19:12:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.472+0000 7f05ec834640 1 -- 172.21.15.28:0/2351232108 wait complete. 2024-04-25T19:12:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.474+0000 7f05ec834640 1 Processor -- start 2024-04-25T19:12:52.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.474+0000 7f05ec834640 1 -- start start 2024-04-25T19:12:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05ec834640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d0004800 0x7f05e80cf680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05ec834640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e81f5ea0 0x7f05e81f9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05ec834640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05e81fa070 0x7f05e80ca550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05ec834640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f05e80d0bf0 con 0x7f05e81f5ea0 2024-04-25T19:12:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05ec834640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f05e80b9400 con 0x7f05d0004800 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05ec834640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f05e80cc950 con 0x7f05e81fa070 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e81f5ea0 0x7f05e81f9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e5d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e81f5ea0 0x7f05e81f9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52276/0 (socket says 172.21.15.28:52276) 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e5d9b640 1 -- 172.21.15.28:0/2071420730 learned_addr learned my addr 172.21.15.28:0/2071420730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e6d9d640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05e81fa070 0x7f05e80ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e659c640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d0004800 0x7f05e80cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e5d9b640 1 -- 172.21.15.28:0/2071420730 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05e81fa070 msgr2=0x7f05e80ca550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e5d9b640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05e81fa070 0x7f05e80ca550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e5d9b640 1 -- 172.21.15.28:0/2071420730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d0004800 msgr2=0x7f05e80cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.475+0000 7f05e5d9b640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d0004800 0x7f05e80cf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.476+0000 7f05e5d9b640 1 -- 172.21.15.28:0/2071420730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e81fa480 con 0x7f05e81f5ea0 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.476+0000 7f05e5d9b640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e81f5ea0 0x7f05e81f9b30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f05dc02adf0 tx=0x7f05e80d13f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.476+0000 7f05d6ffd640 1 -- 172.21.15.28:0/2071420730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc007d10 con 0x7f05e81f5ea0 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.476+0000 7f05d6ffd640 1 -- 172.21.15.28:0/2071420730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05dc042070 con 0x7f05e81f5ea0 2024-04-25T19:12:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.476+0000 7f05d6ffd640 1 -- 172.21.15.28:0/2071420730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05dc004bf0 con 0x7f05e81f5ea0 2024-04-25T19:12:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.476+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e80ce090 con 0x7f05e81f5ea0 2024-04-25T19:12:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.476+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e80c9d50 con 0x7f05e81f5ea0 2024-04-25T19:12:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.477+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e80c9b50 con 0x7f05e81f5ea0 2024-04-25T19:12:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.478+0000 7f05d6ffd640 1 -- 172.21.15.28:0/2071420730 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f05dc03b070 con 0x7f05e81f5ea0 2024-04-25T19:12:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.478+0000 7f05d6ffd640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05c80744e0 0x7f05c80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.478+0000 7f05d6ffd640 1 -- 172.21.15.28:0/2071420730 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f05dc0b4f50 con 0x7f05e81f5ea0 2024-04-25T19:12:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.478+0000 7f05e659c640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05c80744e0 0x7f05c80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.479+0000 7f05e659c640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05c80744e0 0x7f05c80769f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f05d8009a60 tx=0x7f05d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.483+0000 7f05d6ffd640 1 -- 172.21.15.28:0/2071420730 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05e81fa480 con 0x7f05e81f5ea0 2024-04-25T19:12:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.669+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f05e8050740 con 0x7f05e81f5ea0 2024-04-25T19:12:52.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.670+0000 7f05d6ffd640 1 -- 172.21.15.28:0/2071420730 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7f05dc07f550 con 0x7f05e81f5ea0 2024-04-25T19:12:52.672 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:12:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05c80744e0 msgr2=0x7f05c80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05c80744e0 0x7f05c80769f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f05d8009a60 tx=0x7f05d8008040 comp rx=0 tx=0).stop 2024-04-25T19:12:52.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e81f5ea0 msgr2=0x7f05e81f9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e81f5ea0 0x7f05e81f9b30 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f05dc02adf0 tx=0x7f05e80d13f0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 shutdown_connections 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f05c80744e0 0x7f05c80769f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f05e81fa070 0x7f05e80ca550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05e81f5ea0 0x7f05e81f9b30 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 --2- 172.21.15.28:0/2071420730 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05d0004800 0x7f05e80cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 >> 172.21.15.28:0/2071420730 conn(0x7f05e80d2a50 msgr2=0x7f05e80b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 shutdown_connections 2024-04-25T19:12:52.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.672+0000 7f05ec834640 1 -- 172.21.15.28:0/2071420730 wait complete. 2024-04-25T19:12:52.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:12:52.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.800+0000 7f5a57c96640 1 Processor -- start 2024-04-25T19:12:52.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.800+0000 7f5a57c96640 1 -- start start 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.801+0000 7f5a57c96640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5a500b79f0 0x7f5a500b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.801+0000 7f5a57c96640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a50160720 con 0x7f5a500b9920 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.801+0000 7f5a57c96640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a500b8340 con 0x7f5a50160310 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.801+0000 7f5a57c96640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5a500d0f40 con 0x7f5a500b79f0 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.801+0000 7f5a56233640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5a500b9920 0x7f5a500b9d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38386/0 (socket says 172.21.15.28:38386) 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.801+0000 7f5a56233640 1 -- 172.21.15.28:0/238287835 learned_addr learned my addr 172.21.15.28:0/238287835 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.801+0000 7f5a55231640 1 --2- 172.21.15.28:0/238287835 >> v2:172.21.15.53:3300/0 conn(0x7f5a500b79f0 0x7f5a500b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 722111086 0 0) 0x7f5a50160720 con 0x7f5a500b9920 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a3c003520 con 0x7f5a500b9920 2024-04-25T19:12:52.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4293103718 0 0) 0x7f5a3c003520 con 0x7f5a500b9920 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 >> v2:172.21.15.53:3300/0 conn(0x7f5a500b79f0 msgr2=0x7f5a500b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 --2- 172.21.15.28:0/238287835 >> v2:172.21.15.53:3300/0 conn(0x7f5a500b79f0 0x7f5a500b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 >> v1:172.21.15.53:6789/0 conn(0x7f5a50160310 legacy=0x7f5a500d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a500d1220 con 0x7f5a500b9920 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f5a480019f0 con 0x7f5a500b9920 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 >> v1:172.21.15.28:6789/0 conn(0x7f5a500b9920 legacy=0x7f5a500b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c0039b0 0x7f5a3c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3c004800 0x7f5a3c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a3c0051c0 0x7f5a3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a500d0f40 con 0x7f5a3c004800 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5a3c00c160 con 0x7f5a3c0039b0 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5a3c00c360 con 0x7f5a3c0051c0 2024-04-25T19:12:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a55231640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3c004800 0x7f5a3c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a55a32640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c0039b0 0x7f5a3c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a56233640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a3c0051c0 0x7f5a3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a55231640 1 -- 172.21.15.28:0/238287835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a3c0051c0 msgr2=0x7f5a3c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a55231640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a3c0051c0 0x7f5a3c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a55231640 1 -- 172.21.15.28:0/238287835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c0039b0 msgr2=0x7f5a3c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a55231640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c0039b0 0x7f5a3c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a55231640 1 -- 172.21.15.28:0/238287835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a3c003520 con 0x7f5a3c004800 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.802+0000 7f5a56233640 1 -- 172.21.15.28:0/238287835 reap_dead start 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.803+0000 7f5a55231640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3c004800 0x7f5a3c004c10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5a5008dd10 tx=0x7f5a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40bd9280b8c563a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.803+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40036070 con 0x7f5a3c004800 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.803+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a40031400 con 0x7f5a3c004800 2024-04-25T19:12:52.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.803+0000 7f5a54a30640 1 -- 172.21.15.28:0/238287835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40009690 con 0x7f5a3c004800 2024-04-25T19:12:52.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.804+0000 7f5a57c96640 1 -- 172.21.15.28:0/238287835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3c004800 msgr2=0x7f5a3c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.804+0000 7f5a57c96640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3c004800 0x7f5a3c004c10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f5a5008dd10 tx=0x7f5a3c005830 comp rx=0 tx=0).stop 2024-04-25T19:12:52.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.804+0000 7f5a57c96640 1 -- 172.21.15.28:0/238287835 shutdown_connections 2024-04-25T19:12:52.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.804+0000 7f5a57c96640 1 --2- 172.21.15.28:0/238287835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a3c004800 0x7f5a3c004c10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.804+0000 7f5a57c96640 1 -- 172.21.15.28:0/238287835 >> 172.21.15.28:0/238287835 conn(0x7f5a500d2a50 msgr2=0x7f5a500d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.804+0000 7f5a57c96640 1 -- 172.21.15.28:0/238287835 shutdown_connections 2024-04-25T19:12:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.804+0000 7f5a57c96640 1 -- 172.21.15.28:0/238287835 wait complete. 2024-04-25T19:12:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.806+0000 7f5a57c96640 1 Processor -- start 2024-04-25T19:12:52.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.806+0000 7f5a57c96640 1 -- start start 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a57c96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c004800 0x7f5a500ccbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a57c96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a50159900 0x7f5a500cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a57c96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a501f6f00 0x7f5a501581f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a57c96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a500d0bf0 con 0x7f5a501f6f00 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a57c96640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5a500b9400 con 0x7f5a3c004800 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a57c96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5a500cc950 con 0x7f5a50159900 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a56233640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a501f6f00 0x7f5a501581f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a56233640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a501f6f00 0x7f5a501581f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52292/0 (socket says 172.21.15.28:52292) 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a56233640 1 -- 172.21.15.28:0/221481764 learned_addr learned my addr 172.21.15.28:0/221481764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a55231640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a50159900 0x7f5a500cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a56233640 1 -- 172.21.15.28:0/221481764 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a50159900 msgr2=0x7f5a500cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a55a32640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c004800 0x7f5a500ccbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a56233640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a50159900 0x7f5a500cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a56233640 1 -- 172.21.15.28:0/221481764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c004800 msgr2=0x7f5a500ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.807+0000 7f5a56233640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c004800 0x7f5a500ccbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.808+0000 7f5a56233640 1 -- 172.21.15.28:0/221481764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a500ceaf0 con 0x7f5a501f6f00 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.808+0000 7f5a56233640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a501f6f00 0x7f5a501581f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5a500d0980 tx=0x7f5a4800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.808+0000 7f5a467fc640 1 -- 172.21.15.28:0/221481764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4800eb90 con 0x7f5a501f6f00 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.808+0000 7f5a467fc640 1 -- 172.21.15.28:0/221481764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a48018070 con 0x7f5a501f6f00 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.808+0000 7f5a467fc640 1 -- 172.21.15.28:0/221481764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48013420 con 0x7f5a501f6f00 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.808+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a500cedd0 con 0x7f5a501f6f00 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.808+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a500cf390 con 0x7f5a501f6f00 2024-04-25T19:12:52.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.809+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a50050740 con 0x7f5a501f6f00 2024-04-25T19:12:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.809+0000 7f5a467fc640 1 -- 172.21.15.28:0/221481764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5a480135c0 con 0x7f5a501f6f00 2024-04-25T19:12:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.810+0000 7f5a467fc640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5a340744b0 0x7f5a340769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.810+0000 7f5a467fc640 1 -- 172.21.15.28:0/221481764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f5a480957e0 con 0x7f5a501f6f00 2024-04-25T19:12:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.810+0000 7f5a55a32640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5a340744b0 0x7f5a340769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:52.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.810+0000 7f5a55a32640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5a340744b0 0x7f5a340769c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c007fd0 tx=0x7f5a4c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:52.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:52.814+0000 7f5a467fc640 1 -- 172.21.15.28:0/221481764 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5a4805fe00 con 0x7f5a501f6f00 2024-04-25T19:12:53.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.000+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5a5015e9a0 con 0x7f5a501f6f00 2024-04-25T19:12:53.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.000+0000 7f5a467fc640 1 -- 172.21.15.28:0/221481764 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7f5a4805f700 con 0x7f5a501f6f00 2024-04-25T19:12:53.002 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:12:53.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5a340744b0 msgr2=0x7f5a340769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5a340744b0 0x7f5a340769c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c007fd0 tx=0x7f5a4c009040 comp rx=0 tx=0).stop 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a501f6f00 msgr2=0x7f5a501581f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a501f6f00 0x7f5a501581f0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5a500d0980 tx=0x7f5a4800e530 comp rx=0 tx=0).stop 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 shutdown_connections 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5a340744b0 0x7f5a340769c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a501f6f00 0x7f5a501581f0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5a50159900 0x7f5a500cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 --2- 172.21.15.28:0/221481764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a3c004800 0x7f5a500ccbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 >> 172.21.15.28:0/221481764 conn(0x7f5a500d2a50 msgr2=0x7f5a500b9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 shutdown_connections 2024-04-25T19:12:53.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.002+0000 7f5a57c96640 1 -- 172.21.15.28:0/221481764 wait complete. 2024-04-25T19:12:53.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:12:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.127+0000 7f482e734640 1 Processor -- start 2024-04-25T19:12:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.127+0000 7f482e734640 1 -- start start 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f482e734640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482815dfb0 0x7f48280b0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f482e734640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482815e3c0 con 0x7f482806fab0 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f482e734640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48280cf800 con 0x7f48280d0b40 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f482e734640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f48280b09b0 con 0x7f482815dfb0 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f482ccd1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f482806fab0 0x7f48280aea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38402/0 (socket says 172.21.15.28:38402) 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f482ccd1640 1 -- 172.21.15.28:0/276652664 learned_addr learned my addr 172.21.15.28:0/276652664 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f4827fff640 1 --2- 172.21.15.28:0/276652664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482815dfb0 0x7f48280b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f4826ffd640 1 -- 172.21.15.28:0/276652664 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 974124746 0 0) 0x7f482815e3c0 con 0x7f482806fab0 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.129+0000 7f4826ffd640 1 -- 172.21.15.28:0/276652664 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4810003520 con 0x7f482806fab0 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.130+0000 7f4826ffd640 1 -- 172.21.15.28:0/276652664 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3971347028 0 0) 0x7f48280cf800 con 0x7f48280d0b40 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.130+0000 7f4826ffd640 1 -- 172.21.15.28:0/276652664 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f482815e3c0 con 0x7f48280d0b40 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.130+0000 7f4827fff640 1 -- 172.21.15.28:0/276652664 >> v1:172.21.15.53:6789/0 conn(0x7f48280d0b40 legacy=0x7f482806f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.130+0000 7f4827fff640 1 -- 172.21.15.28:0/276652664 >> v1:172.21.15.28:6789/0 conn(0x7f482806fab0 legacy=0x7f48280aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.130+0000 7f4827fff640 1 -- 172.21.15.28:0/276652664 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48280cfaa0 con 0x7f482815dfb0 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.130+0000 7f4827fff640 1 --2- 172.21.15.28:0/276652664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482815dfb0 0x7f48280b0470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f481c0097d0 tx=0x7f481c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f447a917c841f5a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f4826ffd640 1 -- 172.21.15.28:0/276652664 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c037070 con 0x7f482815dfb0 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f4826ffd640 1 -- 172.21.15.28:0/276652664 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c032440 con 0x7f482815dfb0 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f4826ffd640 1 -- 172.21.15.28:0/276652664 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c032810 con 0x7f482815dfb0 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f482e734640 1 -- 172.21.15.28:0/276652664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482815dfb0 msgr2=0x7f48280b0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f482e734640 1 --2- 172.21.15.28:0/276652664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482815dfb0 0x7f48280b0470 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f481c0097d0 tx=0x7f481c02a210 comp rx=0 tx=0).stop 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f482e734640 1 -- 172.21.15.28:0/276652664 shutdown_connections 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f482e734640 1 --2- 172.21.15.28:0/276652664 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482815dfb0 0x7f48280b0470 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f482e734640 1 -- 172.21.15.28:0/276652664 >> 172.21.15.28:0/276652664 conn(0x7f48280742c0 msgr2=0x7f482806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:53.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f482e734640 1 -- 172.21.15.28:0/276652664 shutdown_connections 2024-04-25T19:12:53.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.131+0000 7f482e734640 1 -- 172.21.15.28:0/276652664 wait complete. 2024-04-25T19:12:53.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.133+0000 7f482e734640 1 Processor -- start 2024-04-25T19:12:53.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.133+0000 7f482e734640 1 -- start start 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482e734640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482806fab0 0x7f48281450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482e734640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48280d0b40 0x7f4828145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482e734640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482815dfb0 0x7f4828159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482e734640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48280d0340 con 0x7f482815dfb0 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482e734640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f48280d0060 con 0x7f48280d0b40 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482e734640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f48280d0620 con 0x7f482806fab0 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482815dfb0 0x7f4828159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482815dfb0 0x7f4828159430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52308/0 (socket says 172.21.15.28:52308) 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 -- 172.21.15.28:0/3427433285 learned_addr learned my addr 172.21.15.28:0/3427433285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f48277fe640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48280d0b40 0x7f4828145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f4827fff640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482806fab0 0x7f48281450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 -- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482806fab0 msgr2=0x7f48281450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482806fab0 0x7f48281450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 -- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48280d0b40 msgr2=0x7f4828145bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48280d0b40 0x7f4828145bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 -- 172.21.15.28:0/3427433285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4828158390 con 0x7f482815dfb0 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f4827fff640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482806fab0 0x7f48281450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f48277fe640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48280d0b40 0x7f4828145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f482ccd1640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482815dfb0 0x7f4828159430 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f482000caf0 tx=0x7f482000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.135+0000 7f4824ff9640 1 -- 172.21.15.28:0/3427433285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820017070 con 0x7f482815dfb0 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.136+0000 7f4824ff9640 1 -- 172.21.15.28:0/3427433285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f482000aca0 con 0x7f482815dfb0 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.136+0000 7f4824ff9640 1 -- 172.21.15.28:0/3427433285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820012620 con 0x7f482815dfb0 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.136+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4828158670 con 0x7f482815dfb0 2024-04-25T19:12:53.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.136+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4828158c30 con 0x7f482815dfb0 2024-04-25T19:12:53.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.137+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4828050740 con 0x7f482815dfb0 2024-04-25T19:12:53.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.137+0000 7f4824ff9640 1 -- 172.21.15.28:0/3427433285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f482001a070 con 0x7f482815dfb0 2024-04-25T19:12:53.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.138+0000 7f4824ff9640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48140744b0 0x7f48140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.138+0000 7f4824ff9640 1 -- 172.21.15.28:0/3427433285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f4820095190 con 0x7f482815dfb0 2024-04-25T19:12:53.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.138+0000 7f4827fff640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48140744b0 0x7f48140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.138+0000 7f4827fff640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48140744b0 0x7f48140769c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f481c034000 tx=0x7f481c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:53.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.142+0000 7f4824ff9640 1 -- 172.21.15.28:0/3427433285 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f482005f790 con 0x7f482815dfb0 2024-04-25T19:12:53.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.327+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f482806f720 con 0x7f482815dfb0 2024-04-25T19:12:53.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.327+0000 7f4824ff9640 1 -- 172.21.15.28:0/3427433285 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7f482005f090 con 0x7f482815dfb0 2024-04-25T19:12:53.329 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:12:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48140744b0 msgr2=0x7f48140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48140744b0 0x7f48140769c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f481c034000 tx=0x7f481c02e000 comp rx=0 tx=0).stop 2024-04-25T19:12:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482815dfb0 msgr2=0x7f4828159430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482815dfb0 0x7f4828159430 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f482000caf0 tx=0x7f482000a060 comp rx=0 tx=0).stop 2024-04-25T19:12:53.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 shutdown_connections 2024-04-25T19:12:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f48140744b0 0x7f48140769c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f482815dfb0 0x7f4828159430 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f48280d0b40 0x7f4828145bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 --2- 172.21.15.28:0/3427433285 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f482806fab0 0x7f48281450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 >> 172.21.15.28:0/3427433285 conn(0x7f48280742c0 msgr2=0x7f4828161a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 shutdown_connections 2024-04-25T19:12:53.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.329+0000 7f482e734640 1 -- 172.21.15.28:0/3427433285 wait complete. 2024-04-25T19:12:53.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:12:53.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.453+0000 7f99d6b6c640 1 Processor -- start 2024-04-25T19:12:53.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.453+0000 7f99d6b6c640 1 -- start start 2024-04-25T19:12:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d6b6c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d0160310 0x7f99d00d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d6b6c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f99d00b79f0 0x7f99d00b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d6b6c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99d008cce0 0x7f99d008d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d6b6c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f99d0160720 con 0x7f99d008cce0 2024-04-25T19:12:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d6b6c640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f99d00b8340 con 0x7f99d00b79f0 2024-04-25T19:12:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d6b6c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f99d00d5640 con 0x7f99d0160310 2024-04-25T19:12:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d5109640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99d008cce0 0x7f99d008d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d5109640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99d008cce0 0x7f99d008d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52320/0 (socket says 172.21.15.28:52320) 2024-04-25T19:12:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d5109640 1 -- 172.21.15.28:0/2020857502 learned_addr learned my addr 172.21.15.28:0/2020857502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99cffff640 1 --2- 172.21.15.28:0/2020857502 >> v2:172.21.15.53:3300/0 conn(0x7f99d00b79f0 0x7f99d00b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d4908640 1 --2- 172.21.15.28:0/2020857502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d0160310 0x7f99d00d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d5109640 1 -- 172.21.15.28:0/2020857502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d0160310 msgr2=0x7f99d00d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d5109640 1 --2- 172.21.15.28:0/2020857502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d0160310 0x7f99d00d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d5109640 1 -- 172.21.15.28:0/2020857502 >> v2:172.21.15.53:3300/0 conn(0x7f99d00b79f0 msgr2=0x7f99d00b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d5109640 1 --2- 172.21.15.28:0/2020857502 >> v2:172.21.15.53:3300/0 conn(0x7f99d00b79f0 0x7f99d00b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.455+0000 7f99d5109640 1 -- 172.21.15.28:0/2020857502 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d00d0f40 con 0x7f99d008cce0 2024-04-25T19:12:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.456+0000 7f99d5109640 1 --2- 172.21.15.28:0/2020857502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d008cce0 0x7f99d008d0f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f99d00d6c10 tx=0x7f99c802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bb67ac1a39519c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.456+0000 7f99cf7fe640 1 -- 172.21.15.28:0/2020857502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8036070 con 0x7f99d008cce0 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.456+0000 7f99cf7fe640 1 -- 172.21.15.28:0/2020857502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99c8031600 con 0x7f99d008cce0 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.456+0000 7f99cf7fe640 1 -- 172.21.15.28:0/2020857502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c80319d0 con 0x7f99d008cce0 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 -- 172.21.15.28:0/2020857502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d008cce0 msgr2=0x7f99d008d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/2020857502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d008cce0 0x7f99d008d0f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f99d00d6c10 tx=0x7f99c802a6a0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 -- 172.21.15.28:0/2020857502 shutdown_connections 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/2020857502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d008cce0 0x7f99d008d0f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/2020857502 >> v2:172.21.15.53:3300/0 conn(0x7f99d00b79f0 0x7f99d00b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/2020857502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d0160310 0x7f99d00d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 -- 172.21.15.28:0/2020857502 >> 172.21.15.28:0/2020857502 conn(0x7f99d00d2a50 msgr2=0x7f99d00d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 -- 172.21.15.28:0/2020857502 shutdown_connections 2024-04-25T19:12:53.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.457+0000 7f99d6b6c640 1 -- 172.21.15.28:0/2020857502 wait complete. 2024-04-25T19:12:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.459+0000 7f99d6b6c640 1 Processor -- start 2024-04-25T19:12:53.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.459+0000 7f99d6b6c640 1 -- start start 2024-04-25T19:12:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d6b6c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99d008cce0 0x7f99d01f9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d6b6c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d00b79f0 0x7f99d00ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d6b6c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d0160310 0x7f99d014cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d6b6c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f99d00b9400 con 0x7f99d0160310 2024-04-25T19:12:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d6b6c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f99d00d1500 con 0x7f99d008cce0 2024-04-25T19:12:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d6b6c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f99d00d0bf0 con 0x7f99d00b79f0 2024-04-25T19:12:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d5109640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d0160310 0x7f99d014cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d5109640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d0160310 0x7f99d014cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52328/0 (socket says 172.21.15.28:52328) 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d5109640 1 -- 172.21.15.28:0/3526477478 learned_addr learned my addr 172.21.15.28:0/3526477478 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d4908640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99d008cce0 0x7f99d01f9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99cffff640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d00b79f0 0x7f99d00ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d5109640 1 -- 172.21.15.28:0/3526477478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d00b79f0 msgr2=0x7f99d00ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d5109640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d00b79f0 0x7f99d00ccac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d5109640 1 -- 172.21.15.28:0/3526477478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99d008cce0 msgr2=0x7f99d01f9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d5109640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99d008cce0 0x7f99d01f9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.460+0000 7f99d5109640 1 -- 172.21.15.28:0/3526477478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99d00cc530 con 0x7f99d0160310 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.461+0000 7f99d5109640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d0160310 0x7f99d014cb90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009f40 tx=0x7f99c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.461+0000 7f99cd7fa640 1 -- 172.21.15.28:0/3526477478 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8036050 con 0x7f99d0160310 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.461+0000 7f99cd7fa640 1 -- 172.21.15.28:0/3526477478 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99c8042070 con 0x7f99d0160310 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.461+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99d00cf5d0 con 0x7f99d0160310 2024-04-25T19:12:53.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.461+0000 7f99cd7fa640 1 -- 172.21.15.28:0/3526477478 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c8009510 con 0x7f99d0160310 2024-04-25T19:12:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.461+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99d00cc250 con 0x7f99d0160310 2024-04-25T19:12:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.462+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99d00cf330 con 0x7f99d0160310 2024-04-25T19:12:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.462+0000 7f99cd7fa640 1 -- 172.21.15.28:0/3526477478 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f99c8007bd0 con 0x7f99d0160310 2024-04-25T19:12:53.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.463+0000 7f99cd7fa640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f99bc0744b0 0x7f99bc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.463+0000 7f99d4908640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f99bc0744b0 0x7f99bc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.463+0000 7f99cd7fa640 1 -- 172.21.15.28:0/3526477478 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f99c80b62f0 con 0x7f99d0160310 2024-04-25T19:12:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.463+0000 7f99d4908640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f99bc0744b0 0x7f99bc0769c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f99c0006fd0 tx=0x7f99c0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.468+0000 7f99cd7fa640 1 -- 172.21.15.28:0/3526477478 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99d00cf330 con 0x7f99d0160310 2024-04-25T19:12:53.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.665+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f99d00cd000 con 0x7f99d0160310 2024-04-25T19:12:53.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.666+0000 7f99cd7fa640 1 -- 172.21.15.28:0/3526477478 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) ==== 80+0+0 (secure 0 0 0) 0x7f99c80808e0 con 0x7f99d0160310 2024-04-25T19:12:53.668 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f99bc0744b0 msgr2=0x7f99bc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f99bc0744b0 0x7f99bc0769c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f99c0006fd0 tx=0x7f99c0008040 comp rx=0 tx=0).stop 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d0160310 msgr2=0x7f99d014cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d0160310 0x7f99d014cb90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f99c8009f40 tx=0x7f99c8004a40 comp rx=0 tx=0).stop 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 shutdown_connections 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f99bc0744b0 0x7f99bc0769c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99d0160310 0x7f99d014cb90 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f99d00b79f0 0x7f99d00ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 --2- 172.21.15.28:0/3526477478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99d008cce0 0x7f99d01f9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:53.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 >> 172.21.15.28:0/3526477478 conn(0x7f99d00d2a50 msgr2=0x7f99d00d5870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:53.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 shutdown_connections 2024-04-25T19:12:53.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.668+0000 7f99d6b6c640 1 -- 172.21.15.28:0/3526477478 wait complete. 2024-04-25T19:12:53.681 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T19:12:53.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:53.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T19:12:53.759 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:12:53.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:12:53.760 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T19:12:53.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:12:53.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.932+0000 7f8944651640 1 Processor -- start 2024-04-25T19:12:54.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.932+0000 7f8944651640 1 -- start start 2024-04-25T19:12:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.933+0000 7f8944651640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f893c09caf0 0x7f893c0d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.933+0000 7f8944651640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f893c0d1ce0 con 0x7f893c09cf00 2024-04-25T19:12:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.933+0000 7f8944651640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f893c0cf8d0 con 0x7f893c0d2670 2024-04-25T19:12:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.933+0000 7f8944651640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f893c0b8fa0 con 0x7f893c09caf0 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.933+0000 7f8942bee640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f893c09cf00 0x7f893c0ce720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38414/0 (socket says 172.21.15.28:38414) 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.933+0000 7f8942bee640 1 -- 172.21.15.28:0/1885700341 learned_addr learned my addr 172.21.15.28:0/1885700341 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413713422 0 0) 0x7f893c0d1ce0 con 0x7f893c09cf00 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8928003520 con 0x7f893c09cf00 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89423ed640 1 --2- 172.21.15.28:0/1885700341 >> v2:172.21.15.53:3300/0 conn(0x7f893c09caf0 0x7f893c0d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2401807736 0 0) 0x7f8928003520 con 0x7f893c09cf00 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 >> v2:172.21.15.53:3300/0 conn(0x7f893c09caf0 msgr2=0x7f893c0d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 --2- 172.21.15.28:0/1885700341 >> v2:172.21.15.53:3300/0 conn(0x7f893c09caf0 0x7f893c0d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 >> v1:172.21.15.53:6789/0 conn(0x7f893c0d2670 legacy=0x7f893c0b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893c0cfbb0 con 0x7f893c09cf00 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f8934001a10 con 0x7f893c09cf00 2024-04-25T19:12:54.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89423ed640 1 --2- 172.21.15.28:0/1885700341 >> v2:172.21.15.53:3300/0 conn(0x7f893c09caf0 0x7f893c0d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 >> v1:172.21.15.28:6789/0 conn(0x7f893c09cf00 legacy=0x7f893c0ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89280039b0 0x7f8928003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8928004800 0x7f8928004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8928005300 0x7f892800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f893c0b8fa0 con 0x7f89280039b0 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f892800c2a0 con 0x7f8928004800 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f892800c4a0 con 0x7f8928005300 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89423ed640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89280039b0 0x7f8928003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f8941bec640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8928004800 0x7f8928004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f8942bee640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8928005300 0x7f892800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89423ed640 1 -- 172.21.15.28:0/1885700341 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8928005300 msgr2=0x7f892800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.934+0000 7f89423ed640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8928005300 0x7f892800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.935+0000 7f89423ed640 1 -- 172.21.15.28:0/1885700341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8928004800 msgr2=0x7f8928004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.935+0000 7f89423ed640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8928004800 0x7f8928004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.935+0000 7f89423ed640 1 -- 172.21.15.28:0/1885700341 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8928003520 con 0x7f89280039b0 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.935+0000 7f8942bee640 1 -- 172.21.15.28:0/1885700341 reap_dead start 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.935+0000 7f89423ed640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89280039b0 0x7f8928003e20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f893c069f40 tx=0x7f8928005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409697cfbb385954 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.935+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8938037070 con 0x7f89280039b0 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.935+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893802ac70 con 0x7f89280039b0 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.935+0000 7f89413eb640 1 -- 172.21.15.28:0/1885700341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f893800f040 con 0x7f89280039b0 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.936+0000 7f8944651640 1 -- 172.21.15.28:0/1885700341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89280039b0 msgr2=0x7f8928003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.936+0000 7f8944651640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89280039b0 0x7f8928003e20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f893c069f40 tx=0x7f8928005830 comp rx=0 tx=0).stop 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.936+0000 7f8944651640 1 -- 172.21.15.28:0/1885700341 shutdown_connections 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.936+0000 7f8944651640 1 --2- 172.21.15.28:0/1885700341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89280039b0 0x7f8928003e20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.936+0000 7f8944651640 1 -- 172.21.15.28:0/1885700341 >> 172.21.15.28:0/1885700341 conn(0x7f893c0d03f0 msgr2=0x7f893c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.936+0000 7f8944651640 1 -- 172.21.15.28:0/1885700341 shutdown_connections 2024-04-25T19:12:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.936+0000 7f8944651640 1 -- 172.21.15.28:0/1885700341 wait complete. 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.938+0000 7f8944651640 1 Processor -- start 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.938+0000 7f8944651640 1 -- start start 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8944651640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89280039b0 0x7f893c1dd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8944651640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893c0c9be0 0x7f893c0c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8944651640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f893c0cc130 0x7f893c0cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8944651640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f893c0d4790 con 0x7f893c0c9be0 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8944651640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f893c0b9430 con 0x7f89280039b0 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8944651640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f893c0d2b10 con 0x7f893c0cc130 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8941bec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893c0c9be0 0x7f893c0c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8941bec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893c0c9be0 0x7f893c0c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52352/0 (socket says 172.21.15.28:52352) 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.939+0000 7f8941bec640 1 -- 172.21.15.28:0/3034245117 learned_addr learned my addr 172.21.15.28:0/3034245117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f89423ed640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89280039b0 0x7f893c1dd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8942bee640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f893c0cc130 0x7f893c0cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8941bec640 1 -- 172.21.15.28:0/3034245117 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f893c0cc130 msgr2=0x7f893c0cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8941bec640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f893c0cc130 0x7f893c0cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8941bec640 1 -- 172.21.15.28:0/3034245117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89280039b0 msgr2=0x7f893c1dd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8941bec640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89280039b0 0x7f893c1dd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8941bec640 1 -- 172.21.15.28:0/3034245117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893c0cca80 con 0x7f893c0c9be0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8941bec640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893c0c9be0 0x7f893c0c9ff0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f892c00f820 tx=0x7f892c00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8932ffd640 1 -- 172.21.15.28:0/3034245117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c0050d0 con 0x7f893c0c9be0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8932ffd640 1 -- 172.21.15.28:0/3034245117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f892c018070 con 0x7f893c0c9be0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f893c1e6a50 con 0x7f893c0c9be0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8932ffd640 1 -- 172.21.15.28:0/3034245117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c005490 con 0x7f893c0c9be0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.940+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f893c1e7010 con 0x7f893c0c9be0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.941+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f893c0ca720 con 0x7f893c0c9be0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.942+0000 7f8932ffd640 1 -- 172.21.15.28:0/3034245117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f892c00d730 con 0x7f893c0c9be0 2024-04-25T19:12:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.942+0000 7f8932ffd640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f89200744b0 0x7f89200769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.943+0000 7f8932ffd640 1 -- 172.21.15.28:0/3034245117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f892c0950d0 con 0x7f893c0c9be0 2024-04-25T19:12:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.943+0000 7f89423ed640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f89200744b0 0x7f89200769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.943+0000 7f89423ed640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f89200744b0 0x7f89200769c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8938034000 tx=0x7f893802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:53.947+0000 7f8932ffd640 1 -- 172.21.15.28:0/3034245117 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f893c0cca80 con 0x7f893c0c9be0 2024-04-25T19:12:54.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.170+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f893c050740 con 0x7f893c0c9be0 2024-04-25T19:12:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.170+0000 7f8932ffd640 1 -- 172.21.15.28:0/3034245117 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5169 (secure 0 0 0) 0x7f892c05f080 con 0x7f893c0c9be0 2024-04-25T19:12:54.172 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.172 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/2695669357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2695669357},{"type":"v1","addr":"172.21.15.53:6834","nonce":2695669357}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5310,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6836/2737682189","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2737682189},{"type":"v1","addr":"172.21.15.28:6836","nonce":2737682189}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5313,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/6663801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6835","nonce":6663801},{"type":"v1","addr":"172.21.15.28:6837","nonce":6663801}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:47.076286+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5325},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5325":{"gid":5325,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/803470605","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":803470605},{"type":"v1","addr":"172.21.15.53:6835","nonce":803470605}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5325,"qdb_cluster":[5325]},"id":2}]} 2024-04-25T19:12:54.172 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 18 2024-04-25T19:12:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f89200744b0 msgr2=0x7f89200769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f89200744b0 0x7f89200769c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8938034000 tx=0x7f893802e000 comp rx=0 tx=0).stop 2024-04-25T19:12:54.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893c0c9be0 msgr2=0x7f893c0c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893c0c9be0 0x7f893c0c9ff0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f892c00f820 tx=0x7f892c00fce0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 shutdown_connections 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f89200744b0 0x7f89200769c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f893c0cc130 0x7f893c0cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f893c0c9be0 0x7f893c0c9ff0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 --2- 172.21.15.28:0/3034245117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89280039b0 0x7f893c1dd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 >> 172.21.15.28:0/3034245117 conn(0x7f893c0d03f0 msgr2=0x7f893c1e5e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 shutdown_connections 2024-04-25T19:12:54.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.172+0000 7f8944651640 1 -- 172.21.15.28:0/3034245117 wait complete. 2024-04-25T19:12:54.185 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-04-25T19:12:54.334 INFO:teuthology.orchestra.run.smithi053.stdout:{"message":"","return_code":0} 2024-04-25T19:12:54.334 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-25T19:12:54.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T19:12:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.445+0000 7f7df0d34640 1 Processor -- start 2024-04-25T19:12:54.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.445+0000 7f7df0d34640 1 -- start start 2024-04-25T19:12:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7df0d34640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7dec0d2d10 0x7f7dec0d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7df0d34640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 0x7f7dec0b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7df0d34640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7dec0b4650 0x7f7dec06d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7df0d34640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f7dec0b92a0 con 0x7f7dec0d2d10 2024-04-25T19:12:54.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7df0d34640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f7dec09cea0 con 0x7f7dec0b4650 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7df0d34640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7dec06e8e0 con 0x7f7dec0b3d00 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dea59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7dec0d2d10 0x7f7dec0d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dea59c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7dec0d2d10 0x7f7dec0d21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52362/0 (socket says 172.21.15.28:52362) 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dea59c640 1 -- 172.21.15.28:0/1541863346 learned_addr learned my addr 172.21.15.28:0/1541863346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dead9d640 1 --2- 172.21.15.28:0/1541863346 >> v2:172.21.15.53:3300/0 conn(0x7f7dec0b4650 0x7f7dec06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7de9d9b640 1 --2- 172.21.15.28:0/1541863346 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 0x7f7dec0b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dea59c640 1 -- 172.21.15.28:0/1541863346 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 msgr2=0x7f7dec0b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dea59c640 1 --2- 172.21.15.28:0/1541863346 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 0x7f7dec0b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dea59c640 1 -- 172.21.15.28:0/1541863346 >> v2:172.21.15.53:3300/0 conn(0x7f7dec0b4650 msgr2=0x7f7dec06d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dea59c640 1 --2- 172.21.15.28:0/1541863346 >> v2:172.21.15.53:3300/0 conn(0x7f7dec0b4650 0x7f7dec06d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.447+0000 7f7dea59c640 1 -- 172.21.15.28:0/1541863346 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec0b8ee0 con 0x7f7dec0d2d10 2024-04-25T19:12:54.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.448+0000 7f7dea59c640 1 --2- 172.21.15.28:0/1541863346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0d2d10 0x7f7dec0d21c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7dec08d040 tx=0x7f7de402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed619f8b12e004cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.448+0000 7f7de959a640 1 -- 172.21.15.28:0/1541863346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de4037070 con 0x7f7dec0d2d10 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.448+0000 7f7de959a640 1 -- 172.21.15.28:0/1541863346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de4032440 con 0x7f7dec0d2d10 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.448+0000 7f7de959a640 1 -- 172.21.15.28:0/1541863346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de4032810 con 0x7f7dec0d2d10 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 -- 172.21.15.28:0/1541863346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0d2d10 msgr2=0x7f7dec0d21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 --2- 172.21.15.28:0/1541863346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0d2d10 0x7f7dec0d21c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7dec08d040 tx=0x7f7de402a140 comp rx=0 tx=0).stop 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 -- 172.21.15.28:0/1541863346 shutdown_connections 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 --2- 172.21.15.28:0/1541863346 >> v2:172.21.15.53:3300/0 conn(0x7f7dec0b4650 0x7f7dec06d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 --2- 172.21.15.28:0/1541863346 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 0x7f7dec0b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 --2- 172.21.15.28:0/1541863346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0d2d10 0x7f7dec0d21c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 -- 172.21.15.28:0/1541863346 >> 172.21.15.28:0/1541863346 conn(0x7f7dec0d0450 msgr2=0x7f7dec0d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 -- 172.21.15.28:0/1541863346 shutdown_connections 2024-04-25T19:12:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.449+0000 7f7df0d34640 1 -- 172.21.15.28:0/1541863346 wait complete. 2024-04-25T19:12:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.451+0000 7f7df0d34640 1 Processor -- start 2024-04-25T19:12:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.451+0000 7f7df0d34640 1 -- start start 2024-04-25T19:12:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.452+0000 7f7df0d34640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 0x7f7dec0cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.452+0000 7f7df0d34640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0b4650 0x7f7dec0cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.452+0000 7f7df0d34640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dec0d2d10 0x7f7dec0cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.452+0000 7f7df0d34640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec0d2700 con 0x7f7dec0b4650 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.452+0000 7f7df0d34640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7dec0d3120 con 0x7f7dec0d2d10 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.452+0000 7f7df0d34640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7dec0d2880 con 0x7f7dec0b3d00 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.452+0000 7f7de9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0b4650 0x7f7dec0cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de9d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0b4650 0x7f7dec0cb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52376/0 (socket says 172.21.15.28:52376) 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de9d9b640 1 -- 172.21.15.28:0/4268835092 learned_addr learned my addr 172.21.15.28:0/4268835092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7dea59c640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 0x7f7dec0cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7dead9d640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dec0d2d10 0x7f7dec0cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de9d9b640 1 -- 172.21.15.28:0/4268835092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 msgr2=0x7f7dec0cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de9d9b640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 0x7f7dec0cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de9d9b640 1 -- 172.21.15.28:0/4268835092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dec0d2d10 msgr2=0x7f7dec0cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de9d9b640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dec0d2d10 0x7f7dec0cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de9d9b640 1 -- 172.21.15.28:0/4268835092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dec0cb8d0 con 0x7f7dec0b4650 2024-04-25T19:12:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de9d9b640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0b4650 0x7f7dec0cb050 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400e870 tx=0x7f7dd400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de2ffd640 1 -- 172.21.15.28:0/4268835092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4017070 con 0x7f7dec0b4650 2024-04-25T19:12:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de2ffd640 1 -- 172.21.15.28:0/4268835092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dd4009e40 con 0x7f7dec0b4650 2024-04-25T19:12:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dec0cbbb0 con 0x7f7dec0b4650 2024-04-25T19:12:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7de2ffd640 1 -- 172.21.15.28:0/4268835092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dd400cb20 con 0x7f7dec0b4650 2024-04-25T19:12:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.453+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dec0cc170 con 0x7f7dec0b4650 2024-04-25T19:12:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.454+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dec050740 con 0x7f7dec0b4650 2024-04-25T19:12:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.455+0000 7f7de2ffd640 1 -- 172.21.15.28:0/4268835092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7dd401a070 con 0x7f7dec0b4650 2024-04-25T19:12:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.455+0000 7f7de2ffd640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dcc0744b0 0x7f7dcc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:12:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.456+0000 7f7de2ffd640 1 -- 172.21.15.28:0/4268835092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f7dd40953c0 con 0x7f7dec0b4650 2024-04-25T19:12:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.456+0000 7f7dea59c640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dcc0744b0 0x7f7dcc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:12:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.456+0000 7f7dea59c640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dcc0744b0 0x7f7dcc0769c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7de4034000 tx=0x7f7de402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:12:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.460+0000 7f7de2ffd640 1 -- 172.21.15.28:0/4268835092 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7dd405f400 con 0x7f7dec0b4650 2024-04-25T19:12:54.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.687+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7dec0d0ab0 con 0x7f7dec0b4650 2024-04-25T19:12:54.689 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-25T19:12:54.688+0000 7fa80f29e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:12:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.689+0000 7f7de2ffd640 1 -- 172.21.15.28:0/4268835092 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v19) ==== 161+0+0 (secure 0 0 0) 0x7f7dd4063700 con 0x7f7dec0b4650 2024-04-25T19:12:54.691 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T19:12:54.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.692+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dcc0744b0 msgr2=0x7f7dcc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.692+0000 7f7df0d34640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dcc0744b0 0x7f7dcc0769c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7de4034000 tx=0x7f7de402e000 comp rx=0 tx=0).stop 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.692+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0b4650 msgr2=0x7f7dec0cb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.692+0000 7f7df0d34640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0b4650 0x7f7dec0cb050 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400e870 tx=0x7f7dd400ed30 comp rx=0 tx=0).stop 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.693+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 shutdown_connections 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.693+0000 7f7df0d34640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7dcc0744b0 0x7f7dcc0769c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.693+0000 7f7df0d34640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7dec0d2d10 0x7f7dec0cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.693+0000 7f7df0d34640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7dec0b4650 0x7f7dec0cb050 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.693+0000 7f7df0d34640 1 --2- 172.21.15.28:0/4268835092 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7dec0b3d00 0x7f7dec0cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.693+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 >> 172.21.15.28:0/4268835092 conn(0x7f7dec0d0450 msgr2=0x7f7dec071560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:12:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.693+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 shutdown_connections 2024-04-25T19:12:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:12:54.693+0000 7f7df0d34640 1 -- 172.21.15.28:0/4268835092 wait complete. 2024-04-25T19:12:54.697 INFO:tasks.ceph.mds.b.smithi053.stderr: -4391> 2024-04-25T19:12:48.652+0000 7efe21111640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T19:12:54.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T19:12:54.709 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T19:12:54.749 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-04-25T19:12:54.750 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.751 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.752 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]" 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-04-25T19:12:54.753 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-04-25T19:12:54.754 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug trim objects = true 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-04-25T19:12:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 20 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 20 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 10 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T19:12:54.756 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:12:54.757 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:12:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.759 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: mon down mkfs grace = 300 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T19:12:54.760 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-04-25T19:12:54.761 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = True 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = False 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-04-25T19:12:54.762 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-04-25T19:12:54.763 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-04-25T19:12:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-04-25T19:12:54.765 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-04-25T19:12:54.766 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "172.21.15.28,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]" 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on skipped message = False 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:12:54.767 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-04-25T19:12:54.768 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug trim objects = true 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.769 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 20 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 20 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 10 2024-04-25T19:12:54.770 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-04-25T19:12:54.771 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:12:54.772 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.773 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: mon down mkfs grace = 300 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: mon op complaint time = 120 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T19:12:54.774 INFO:teuthology.orchestra.run.smithi053.stdout: client mount timeout = 600 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = "" 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a] 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c] 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b] 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d] 2024-04-25T19:12:54.775 INFO:teuthology.orchestra.run.smithi053.stdout:[mds] 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds = 20 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds balancer = 20 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug frag = True 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug scatterstat = True 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: mds op complaint time = 180 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: mds verify scatter = False 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-04-25T19:12:54.776 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-04-25T19:12:54.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T19:12:54.820 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T19:12:54.860 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-25T19:12:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-04-25T19:12:54.860 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T19:12:54.861 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-04-25T19:12:54.862 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-04-25T19:12:54.863 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]" 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-04-25T19:12:54.864 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug trim objects = true 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-04-25T19:12:54.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 20 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 20 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 10 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-04-25T19:12:54.866 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.867 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T19:12:54.868 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: mon down mkfs grace = 300 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T19:12:54.869 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-04-25T19:12:54.870 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = False 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = False 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-04-25T19:12:54.871 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.872 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-04-25T19:12:54.873 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-04-25T19:12:54.874 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "172.21.15.28,172.21.15.53,[v2:172.21.15.53:3301,v1:172.21.15.53:6790]" 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on skipped message = False 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-04-25T19:12:54.875 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug trim objects = true 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-04-25T19:12:54.876 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 20 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 20 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 10 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2024-04-25T19:12:54.877 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-04-25T19:12:54.878 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T19:12:54.879 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: mon down mkfs grace = 300 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: mon op complaint time = 120 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-04-25T19:12:54.880 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout: client mount timeout = 600 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = "" 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a] 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c] 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b] 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d] 2024-04-25T19:12:54.881 INFO:teuthology.orchestra.run.smithi053.stdout:[mds] 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds = 20 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds balancer = 20 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug frag = True 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug scatterstat = False 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: mds op complaint time = 180 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: mds verify scatter = False 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-04-25T19:12:54.882 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-04-25T19:12:54.884 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T19:12:54.884 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T19:12:54.884 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T19:12:54.884 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:12:54.884 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T19:12:54.884 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T19:12:54.884 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T19:12:54.885 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:12:54.885 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T19:12:54.885 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T19:12:54.885 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T19:12:54.885 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:12:54.885 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T19:12:54.885 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T19:12:54.885 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T19:12:54.886 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:12:54.936 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T19:12:54.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T19:12:54.937 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T19:12:54.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T19:12:54.938 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T19:12:54.938 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T19:12:54.978 INFO:tasks.ceph.mds.a:Started 2024-04-25T19:12:54.978 DEBUG:teuthology.parallel:result is None 2024-04-25T19:12:54.980 INFO:tasks.ceph.mds.d:Started 2024-04-25T19:12:54.980 DEBUG:teuthology.parallel:result is None 2024-04-25T19:12:54.980 INFO:tasks.ceph.mds.b:Started 2024-04-25T19:12:54.981 DEBUG:teuthology.parallel:result is None 2024-04-25T19:12:55.062 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-04-25T19:12:55.065 INFO:tasks.ceph.mds.b.smithi053.stdout:starting mds.b at 2024-04-25T19:12:55.065 INFO:tasks.ceph.mds.d.smithi053.stdout:starting mds.d at 2024-04-25T19:12:55.545 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-04-25T19:13:00.935 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T19:13:00.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T19:13:00.939 INFO:tasks.ceph.mds.c:Started 2024-04-25T19:13:00.939 DEBUG:teuthology.parallel:result is None 2024-04-25T19:13:00.940 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph purge cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T19:13:01.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.001+0000 7ff2d3d0ff40 1 Processor -- start 2024-04-25T19:13:01.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.001+0000 7ff2d3d0ff40 1 -- start start 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2d3d0ff40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a310 0x55e29112baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2d3d0ff40 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e29114a720 0x55e2911cf870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2d3d0ff40 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e2911d77d0 0x55e2911d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2d3d0ff40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2910a8e40 con 0x55e29114a310 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2d3d0ff40 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e2910ace80 con 0x55e2911d77d0 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2d3d0ff40 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55e2911da3c0 con 0x55e29114a720 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cd7d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a310 0x55e29112baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cd7d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a310 0x55e29112baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51008/0 (socket says 172.21.15.28:51008) 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/2670428172 learned_addr learned my addr 172.21.15.28:0/2670428172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e29114a720 0x55e2911cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e2911d77d0 0x55e2911d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e29114a720 msgr2=0x55e2911cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e29114a720 0x55e2911cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e2911d77d0 msgr2=0x55e2911d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e2911d77d0 0x55e2911d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/2670428172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e291149240 con 0x55e29114a310 2024-04-25T19:13:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e29114a720 0x55e2911cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.002+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e2911d77d0 0x55e2911d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.003+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a310 0x55e29112baa0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0001e00 tx=0x7ff2c000b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7c86279ea9eb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.003+0000 7ff2c77fe640 1 -- 172.21.15.28:0/2670428172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0016070 con 0x55e29114a310 2024-04-25T19:13:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.003+0000 7ff2c77fe640 1 -- 172.21.15.28:0/2670428172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c000d6d0 con 0x55e29114a310 2024-04-25T19:13:01.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.003+0000 7ff2c77fe640 1 -- 172.21.15.28:0/2670428172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c000db00 con 0x55e29114a310 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/2670428172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a310 msgr2=0x55e29112baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a310 0x55e29112baa0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0001e00 tx=0x7ff2c000b650 comp rx=0 tx=0).stop 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/2670428172 shutdown_connections 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e2911d77d0 0x55e2911d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e29114a720 0x55e2911cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/2670428172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a310 0x55e29112baa0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/2670428172 >> 172.21.15.28:0/2670428172 conn(0x55e291024ac0 msgr2=0x55e2911d11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/2670428172 shutdown_connections 2024-04-25T19:13:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.004+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/2670428172 wait complete. 2024-04-25T19:13:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.005+0000 7ff2d3d0ff40 1 Processor -- start 2024-04-25T19:13:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.005+0000 7ff2d3d0ff40 1 -- start start 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2d3d0ff40 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e29114a310 0x55e291103bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2d3d0ff40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a720 0x55e2911c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2d3d0ff40 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2911d77d0 0x55e2911c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2d3d0ff40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e2911d03d0 con 0x55e29114a720 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2d3d0ff40 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55e291149520 con 0x55e29114a310 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2d3d0ff40 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55e2911c9d80 con 0x55e2911d77d0 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2ccfd8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a720 0x55e2911c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2ccfd8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a720 0x55e2911c6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51022/0 (socket says 172.21.15.28:51022) 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.006+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 learned_addr learned my addr 172.21.15.28:0/1555377528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2911d77d0 0x55e2911c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e29114a310 0x55e291103bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2911d77d0 msgr2=0x55e2911c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55e2911d77d0 0x55e2911c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e29114a310 msgr2=0x55e291103bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55e29114a310 0x55e291103bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2911c75d0 con 0x55e29114a720 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a720 0x55e2911c6600 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8045e70 tx=0x7ff2b8002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2c67fc640 1 -- 172.21.15.28:0/1555377528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8066070 con 0x55e29114a720 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2c67fc640 1 -- 172.21.15.28:0/1555377528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2b8055460 con 0x55e29114a720 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55e2911cd8e0 con 0x55e29114a720 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.007+0000 7ff2c67fc640 1 -- 172.21.15.28:0/1555377528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2b805a040 con 0x55e29114a720 2024-04-25T19:13:01.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.008+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2911c78b0 con 0x55e29114a720 2024-04-25T19:13:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.009+0000 7ff2c67fc640 1 -- 172.21.15.28:0/1555377528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff2b8069070 con 0x55e29114a720 2024-04-25T19:13:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.010+0000 7ff2c67fc640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff2b0072e90 0x7ff2b00753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.010+0000 7ff2c67fc640 1 -- 172.21.15.28:0/1555377528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6187+0+0 (secure 0 0 0) 0x7ff2b80d7a10 con 0x55e29114a720 2024-04-25T19:13:01.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.010+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff2b0072e90 0x7ff2b00753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.012 INFO:teuthology.orchestra.run.smithi028.stdout:Warning: using slow linear search 2024-04-25T19:13:01.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.010+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55e2911040f0 0x55e2911c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.010+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:1 5.0 5:00000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x55e2911040f0 2024-04-25T19:13:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.010+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55e2911040f0 0x55e2911c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.011+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff2b0072e90 0x7ff2b00753a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0001e00 tx=0x7ff2c0002000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.011+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55e2911040f0 0x55e2911c8e80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.011+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== osd_op_reply(1 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.011+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7ff2c80138b0 0x7ff2c8012330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.011+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:2 5.10 5:08000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c80138b0 2024-04-25T19:13:01.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.012+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7ff2c80138b0 0x7ff2c8012330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.012+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7ff2c80138b0 0x7ff2c8012330 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.013+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 1 ==== osd_op_reply(2 [pgnls start_epoch 42 out=177b] v38'10 uv10 ondisk = 1) ==== 144+0+177 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.013+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7ff2c001f070 0x7ff2c0010440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.013+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:3 5.8 5:10000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c001f070 2024-04-25T19:13:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.013+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7ff2c001f070 0x7ff2c0010440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.013+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7ff2c001f070 0x7ff2c0010440 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.014+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.2 v2:172.21.15.28:6816/3410596961 1 ==== osd_op_reply(3 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c001f070 2024-04-25T19:13:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.014+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7ff2b8022b00 0x7ff2b80a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.014+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:4 5.18 5:18000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b8022b00 2024-04-25T19:13:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.014+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7ff2b8022b00 0x7ff2b80a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.015+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7ff2b8022b00 0x7ff2b80a06e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.015+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.4 v2:172.21.15.53:6824/2055146181 1 ==== osd_op_reply(4 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b8022b00 2024-04-25T19:13:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.015+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:5 5.4 5:20000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c80138b0 2024-04-25T19:13:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.016+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 2 ==== osd_op_reply(5 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.016+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:6 5.14 5:28000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x55e2911040f0 2024-04-25T19:13:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.016+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 2 ==== osd_op_reply(6 [pgnls start_epoch 42 out=149b] v38'19 uv19 ondisk = 1) ==== 144+0+149 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.016+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7ff2c8016930 0x7ff2c80112c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.016+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:7 5.c 5:30000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c8016930 2024-04-25T19:13:01.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.016+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7ff2c8016930 0x7ff2c80112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.017+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7ff2c8016930 0x7ff2c80112c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.017+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.1 v2:172.21.15.28:6808/1898594658 1 ==== osd_op_reply(7 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c8016930 2024-04-25T19:13:01.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.017+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:8 5.1c 5:38000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b8022b00 2024-04-25T19:13:01.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.018+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.4 v2:172.21.15.53:6824/2055146181 2 ==== osd_op_reply(8 [pgnls start_epoch 42 out=73b] v37'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b8022b00 2024-04-25T19:13:01.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.018+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7ff2c80243f0 0x7ff2c8024800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.018+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:9 5.2 5:40000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c80243f0 2024-04-25T19:13:01.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.018+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7ff2c80243f0 0x7ff2c8024800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.019+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7ff2c80243f0 0x7ff2c8024800 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.019+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== osd_op_reply(9 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c80243f0 2024-04-25T19:13:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.019+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:10 5.12 5:48000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c8016930 2024-04-25T19:13:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.020+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.1 v2:172.21.15.28:6808/1898594658 2 ==== osd_op_reply(10 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c8016930 2024-04-25T19:13:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.020+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:11 5.a 5:50000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c001f070 2024-04-25T19:13:01.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.020+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.2 v2:172.21.15.28:6816/3410596961 2 ==== osd_op_reply(11 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c001f070 2024-04-25T19:13:01.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.020+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:12 5.1a 5:58000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c80138b0 2024-04-25T19:13:01.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.020+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 3 ==== osd_op_reply(12 [pgnls start_epoch 42 out=49b] v37'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.020+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:13 5.6 5:60000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c001f070 2024-04-25T19:13:01.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.021+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.2 v2:172.21.15.28:6816/3410596961 3 ==== osd_op_reply(13 [pgnls start_epoch 42 out=77b] v38'4 uv4 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c001f070 2024-04-25T19:13:01.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.021+0000 7ff2ccfd8640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7ff2b801f480 0x7ff2b8055810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.021+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:14 5.16 5:68000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b801f480 2024-04-25T19:13:01.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.021+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7ff2b801f480 0x7ff2b8055810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.022+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7ff2b801f480 0x7ff2b8055810 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.022+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 1 ==== osd_op_reply(14 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.022+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:15 5.e 5:70000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b8022b00 2024-04-25T19:13:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.023+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.4 v2:172.21.15.53:6824/2055146181 3 ==== osd_op_reply(15 [pgnls start_epoch 42 out=73b] v37'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b8022b00 2024-04-25T19:13:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.023+0000 7ff2cdfda640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7ff2c8023380 0x7ff2c8023790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.023+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:16 5.1e 5:78000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c8023380 2024-04-25T19:13:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.023+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7ff2c8023380 0x7ff2c8023790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.023+0000 7ff2cd7d9640 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7ff2c8023380 0x7ff2c8023790 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.024+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== osd_op_reply(16 [pgnls start_epoch 42 out=49b] v37'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c8023380 2024-04-25T19:13:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.024+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:17 5.1 5:80000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b8022b00 2024-04-25T19:13:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.024+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.4 v2:172.21.15.53:6824/2055146181 4 ==== osd_op_reply(17 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b8022b00 2024-04-25T19:13:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.024+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:18 5.11 5:88000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c80243f0 2024-04-25T19:13:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.025+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.6 v2:172.21.15.53:6809/3653668189 2 ==== osd_op_reply(18 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c80243f0 2024-04-25T19:13:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.025+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:19 5.9 5:90000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c80138b0 2024-04-25T19:13:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.025+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 4 ==== osd_op_reply(19 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.025+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:20 5.19 5:98000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c8016930 2024-04-25T19:13:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.026+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.1 v2:172.21.15.28:6808/1898594658 3 ==== osd_op_reply(20 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c8016930 2024-04-25T19:13:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.026+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:21 5.5 5:a0000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c8023380 2024-04-25T19:13:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.026+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.0 v2:172.21.15.28:6800/1137456103 2 ==== osd_op_reply(21 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c8023380 2024-04-25T19:13:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.026+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:22 5.15 5:a8000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b801f480 2024-04-25T19:13:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.027+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 2 ==== osd_op_reply(22 [pgnls start_epoch 42 out=97b] v37'3 uv3 ondisk = 1) ==== 144+0+97 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.027+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:23 5.d 5:b0000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c001f070 2024-04-25T19:13:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.027+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.2 v2:172.21.15.28:6816/3410596961 4 ==== osd_op_reply(23 [pgnls start_epoch 42 out=74b] v37'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c001f070 2024-04-25T19:13:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.027+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:24 5.1d 5:b8000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c8016930 2024-04-25T19:13:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.027+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.1 v2:172.21.15.28:6808/1898594658 4 ==== osd_op_reply(24 [pgnls start_epoch 42 out=73b] v37'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c8016930 2024-04-25T19:13:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.027+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:25 5.3 5:c0000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c8023380 2024-04-25T19:13:01.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.028+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.0 v2:172.21.15.28:6800/1137456103 3 ==== osd_op_reply(25 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c8023380 2024-04-25T19:13:01.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.028+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:26 5.13 5:c8000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x55e2911040f0 2024-04-25T19:13:01.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.028+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 3 ==== osd_op_reply(26 [pgnls start_epoch 42 out=98b] v38'3 uv3 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.028+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:27 5.b 5:d0000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c001f070 2024-04-25T19:13:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.028+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.2 v2:172.21.15.28:6816/3410596961 5 ==== osd_op_reply(27 [pgnls start_epoch 42 out=76b] v38'3 uv3 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c001f070 2024-04-25T19:13:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.029+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:28 5.1b 5:d8000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b801f480 2024-04-25T19:13:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.029+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 3 ==== osd_op_reply(28 [pgnls start_epoch 42 out=49b] v37'1 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.029+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:29 5.7 5:e0000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b801f480 2024-04-25T19:13:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.029+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 4 ==== osd_op_reply(29 [pgnls start_epoch 42 out=73b] v37'2 uv2 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.029+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:30 5.17 5:e8000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x55e2911040f0 2024-04-25T19:13:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.030+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 4 ==== osd_op_reply(30 [pgnls start_epoch 42 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.030+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:31 5.f 5:f0000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2b801f480 2024-04-25T19:13:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.030+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 5 ==== osd_op_reply(31 [pgnls start_epoch 42 out=73b] v37'1 uv1 ondisk = 1) ==== 144+0+73 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.030+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:32 5.1f 5:f8000000:%01:::head [pgnls start_epoch 42 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e42) -- 0x55e2911c93c0 con 0x7ff2c80243f0 2024-04-25T19:13:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.6 v2:172.21.15.53:6809/3653668189 3 ==== osd_op_reply(32 [pgnls start_epoch 42 out=101b] v38'6 uv6 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c80243f0 2024-04-25T19:13:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:33 5.10 5:09401a81:::601.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e29127e730 con 0x7ff2c80138b0 2024-04-25T19:13:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:34 5.10 5:0bd6d154:::602.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291281000 con 0x7ff2c80138b0 2024-04-25T19:13:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:35 5.10 5:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291281850 con 0x7ff2c80138b0 2024-04-25T19:13:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:36 5.10 5:0d82a743:::600.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e2912820a0 con 0x7ff2c80138b0 2024-04-25T19:13:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:37 5.10 5:0d89b25e:::603.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e2912828f0 con 0x7ff2c80138b0 2024-04-25T19:13:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:38 5.14 5:287034c4:::1.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291283140 con 0x55e2911040f0 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:39 5.14 5:292cf221:::200.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291283990 con 0x55e2911040f0 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:40 5.14 5:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291284330 con 0x55e2911040f0 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:41 5.14 5:2f55791f:::606.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291284cd0 con 0x55e2911040f0 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:42 5.1c 5:3ed09add:::607.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291285790 con 0x7ff2b8022b00 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:43 5.6 5:654134d2:::mds0_openfiles.0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291286310 con 0x7ff2c001f070 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:44 5.e 5:717a0223:::608.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291286e90 con 0x7ff2b8022b00 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:45 5.15 5:a93a17c2:::604.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291287a00 con 0x7ff2b801f480 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:46 5.15 5:aa448500:::500.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291288520 con 0x7ff2b801f480 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:47 5.d 5:b50e409b:::mds_snaptable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291289070 con 0x7ff2c001f070 2024-04-25T19:13:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.031+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:48 5.1d 5:b871830b:::605.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291289ba0 con 0x7ff2c8016930 2024-04-25T19:13:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.033+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.2 v2:172.21.15.28:6816/3410596961 6 ==== osd_op_reply(43 mds0_openfiles.0 [delete] v42'5 uv5 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c001f070 2024-04-25T19:13:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.033+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:49 5.13 5:c91d4a1d:::mds0_inotable:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291289fc0 con 0x55e2911040f0 2024-04-25T19:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.034+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.2 v2:172.21.15.28:6816/3410596961 7 ==== osd_op_reply(47 mds_snaptable [delete] v42'2 uv2 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c001f070 2024-04-25T19:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.034+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 6 ==== osd_op_reply(45 604.00000000 [delete] v42'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.034+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.4 v2:172.21.15.53:6824/2055146181 5 ==== osd_op_reply(42 607.00000000 [delete] v42'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b8022b00 2024-04-25T19:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.034+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:50 5.13 5:cd5a64a3:::100.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291287430 con 0x55e2911040f0 2024-04-25T19:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.034+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.4 v2:172.21.15.53:6824/2055146181 6 ==== osd_op_reply(44 608.00000000 [delete] v42'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b8022b00 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.034+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 5 ==== osd_op_reply(38 1.00000000.inode [delete] v42'20 uv20 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.034+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:51 5.b 5:d0630e4c:::mds0_sessionmap:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e291287430 con 0x7ff2c001f070 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:52 5.7 5:e0b41d06:::609.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e2912868c0 con 0x7ff2b801f480 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:53 5.f 5:f1f69726:::400.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e29128a470 con 0x7ff2b801f480 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:54 5.1f 5:fe07bba1:::100.00000000.inode:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e29128acc0 con 0x7ff2c80243f0 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 6 ==== osd_op_reply(39 200.00000000 [delete] v42'21 uv21 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 7 ==== osd_op_reply(46 500.00000000 [delete] v42'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:55 5.1f 5:ff5b34d6:::1.00000000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e42) -- 0x55e2912868c0 con 0x7ff2c80243f0 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 5 ==== osd_op_reply(33 601.00000000 [delete] v42'11 uv11 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 6 ==== osd_op_reply(34 602.00000000 [delete] v42'12 uv12 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 7 ==== osd_op_reply(35 10000000000.00000000 [delete] v42'13 uv13 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.1 v2:172.21.15.28:6808/1898594658 5 ==== osd_op_reply(48 605.00000000 [delete] v42'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c8016930 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 7 ==== osd_op_reply(40 200.00000001 [delete] v42'22 uv22 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 8 ==== osd_op_reply(41 606.00000000 [delete] v42'23 uv23 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 9 ==== osd_op_reply(49 mds0_inotable [delete] v42'4 uv4 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.035+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 8 ==== osd_op_reply(36 600.00000000 [delete] v42'14 uv14 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.036+0000 7ff2cd7d9640 1 -- 172.21.15.28:0/1555377528 <== osd.7 v2:172.21.15.53:6808/79451145 9 ==== osd_op_reply(37 603.00000000 [delete] v42'15 uv15 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c000d9d0 con 0x7ff2c80138b0 2024-04-25T19:13:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.036+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.3 v2:172.21.15.28:6818/1928233777 10 ==== osd_op_reply(50 100.00000000 [delete] v42'5 uv5 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x55e2911040f0 2024-04-25T19:13:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.036+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.2 v2:172.21.15.28:6816/3410596961 8 ==== osd_op_reply(51 mds0_sessionmap [delete] v42'4 uv4 ondisk = 0) ==== 159+0+0 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c001f070 2024-04-25T19:13:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.6 v2:172.21.15.53:6809/3653668189 4 ==== osd_op_reply(54 100.00000000.inode [delete] v42'7 uv7 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c80243f0 2024-04-25T19:13:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 8 ==== osd_op_reply(52 609.00000000 [delete] v42'3 uv3 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 <== osd.5 v2:172.21.15.53:6800/2342659583 9 ==== osd_op_reply(53 400.00000000 [delete] v42'2 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x7ff2c80018e0 con 0x7ff2b801f480 2024-04-25T19:13:01.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2ccfd8640 1 -- 172.21.15.28:0/1555377528 <== osd.6 v2:172.21.15.53:6809/3653668189 5 ==== osd_op_reply(55 1.00000000 [delete] v42'8 uv8 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x7ff2b8072070 con 0x7ff2c80243f0 2024-04-25T19:13:01.040 INFO:teuthology.orchestra.run.smithi028.stdout:Removed 23 objects 2024-04-25T19:13:01.040 INFO:teuthology.orchestra.run.smithi028.stdout:successfully purged pool cephfs_metadata 2024-04-25T19:13:01.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7ff2c8023380 msgr2=0x7ff2c8023790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7ff2c8023380 0x7ff2c8023790 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7ff2c8016930 msgr2=0x7ff2c80112c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7ff2c8016930 0x7ff2c80112c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7ff2c001f070 msgr2=0x7ff2c0010440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7ff2c001f070 0x7ff2c0010440 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55e2911040f0 msgr2=0x55e2911c8e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55e2911040f0 0x55e2911c8e80 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 reap_dead start 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.037+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7ff2b8022b00 msgr2=0x7ff2b80a06e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7ff2b8022b00 0x7ff2b80a06e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 reap_dead start 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7ff2b801f480 msgr2=0x7ff2b8055810 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7ff2b801f480 0x7ff2b8055810 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7ff2c80243f0 msgr2=0x7ff2c8024800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7ff2c80243f0 0x7ff2c8024800 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7ff2c80138b0 msgr2=0x7ff2c8012330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7ff2c80138b0 0x7ff2c8012330 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff2b0072e90 msgr2=0x7ff2b00753a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7ff2b0072e90 0x7ff2b00753a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0001e00 tx=0x7ff2c0002000 comp rx=0 tx=0).stop 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a720 msgr2=0x55e2911c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 --2- 172.21.15.28:0/1555377528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e29114a720 0x55e2911c6600 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8045e70 tx=0x7ff2b8002fb0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2cdfda640 1 -- 172.21.15.28:0/1555377528 reap_dead start 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 shutdown_connections 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 >> 172.21.15.28:0/1555377528 conn(0x55e291024ac0 msgr2=0x55e2911cf870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 shutdown_connections 2024-04-25T19:13:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.038+0000 7ff2d3d0ff40 1 -- 172.21.15.28:0/1555377528 wait complete. 2024-04-25T19:13:01.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:01.045 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-04-25T19:13:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.156+0000 7efd678d9640 1 Processor -- start 2024-04-25T19:13:01.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.156+0000 7efd678d9640 1 -- start start 2024-04-25T19:13:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd678d9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 0x7efd6015dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd678d9640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efd600b7660 0x7efd600b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd678d9640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd60161210 con 0x7efd601606f0 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd678d9640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7efd600b7fb0 con 0x7efd600b7660 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd678d9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efd601614a0 con 0x7efd60164990 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd64e74640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efd600b7660 0x7efd600b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd64e74640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efd600b7660 0x7efd600b7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51040/0 (socket says 172.21.15.28:51040) 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd64e74640 1 -- 172.21.15.28:0/4287649269 learned_addr learned my addr 172.21.15.28:0/4287649269 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.157+0000 7efd65675640 1 --2- 172.21.15.28:0/4287649269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 0x7efd6015dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.158+0000 7efd64e74640 1 -- 172.21.15.28:0/4287649269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 msgr2=0x7efd6015dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.158+0000 7efd64e74640 1 --2- 172.21.15.28:0/4287649269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 0x7efd6015dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.158+0000 7efd64e74640 1 -- 172.21.15.28:0/4287649269 >> v1:172.21.15.53:6789/0 conn(0x7efd601606f0 legacy=0x7efd60160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.158+0000 7efd64e74640 1 -- 172.21.15.28:0/4287649269 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd60163eb0 con 0x7efd600b7660 2024-04-25T19:13:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.158+0000 7efd64e74640 1 --2- 172.21.15.28:0/4287649269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd600b7a70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efd50009870 tx=0x7efd5002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903812a7e1c3bc0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd57fff640 1 -- 172.21.15.28:0/4287649269 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50037070 con 0x7efd600b7660 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd57fff640 1 -- 172.21.15.28:0/4287649269 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd50032440 con 0x7efd600b7660 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd57fff640 1 -- 172.21.15.28:0/4287649269 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd50032810 con 0x7efd600b7660 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd678d9640 1 -- 172.21.15.28:0/4287649269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 msgr2=0x7efd600b7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd678d9640 1 --2- 172.21.15.28:0/4287649269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd600b7a70 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7efd50009870 tx=0x7efd5002a270 comp rx=0 tx=0).stop 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd678d9640 1 -- 172.21.15.28:0/4287649269 shutdown_connections 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd678d9640 1 --2- 172.21.15.28:0/4287649269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd600b7a70 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd678d9640 1 --2- 172.21.15.28:0/4287649269 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 0x7efd6015dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd678d9640 1 -- 172.21.15.28:0/4287649269 >> 172.21.15.28:0/4287649269 conn(0x7efd600d2840 msgr2=0x7efd600d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd678d9640 1 -- 172.21.15.28:0/4287649269 shutdown_connections 2024-04-25T19:13:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.159+0000 7efd678d9640 1 -- 172.21.15.28:0/4287649269 wait complete. 2024-04-25T19:13:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.161+0000 7efd678d9640 1 Processor -- start 2024-04-25T19:13:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.161+0000 7efd678d9640 1 -- start start 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.162+0000 7efd678d9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd6015b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.162+0000 7efd678d9640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd601606f0 0x7efd600b5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.162+0000 7efd678d9640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 0x7efd600b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.162+0000 7efd678d9640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd600b45d0 con 0x7efd600b7660 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.162+0000 7efd678d9640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efd60164470 con 0x7efd601606f0 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.162+0000 7efd678d9640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efd600b4750 con 0x7efd60164990 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.162+0000 7efd65675640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd6015b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65675640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd6015b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51052/0 (socket says 172.21.15.28:51052) 2024-04-25T19:13:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65675640 1 -- 172.21.15.28:0/60051944 learned_addr learned my addr 172.21.15.28:0/60051944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd64e74640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd601606f0 0x7efd600b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65e76640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 0x7efd600b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65675640 1 -- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 msgr2=0x7efd600b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65675640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 0x7efd600b5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65675640 1 -- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd601606f0 msgr2=0x7efd600b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65675640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd601606f0 0x7efd600b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65675640 1 -- 172.21.15.28:0/60051944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd601ea010 con 0x7efd600b7660 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd64e74640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd601606f0 0x7efd600b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd65675640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd6015b8d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00e840 tx=0x7efd5c00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd55ffb640 1 -- 172.21.15.28:0/60051944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c017070 con 0x7efd600b7660 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd55ffb640 1 -- 172.21.15.28:0/60051944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd5c00c8e0 con 0x7efd600b7660 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd601ea2f0 con 0x7efd600b7660 2024-04-25T19:13:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd55ffb640 1 -- 172.21.15.28:0/60051944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5c00ccb0 con 0x7efd600b7660 2024-04-25T19:13:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.163+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd601ea8b0 con 0x7efd600b7660 2024-04-25T19:13:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.164+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd601ec710 con 0x7efd600b7660 2024-04-25T19:13:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.165+0000 7efd55ffb640 1 -- 172.21.15.28:0/60051944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efd5c01a070 con 0x7efd600b7660 2024-04-25T19:13:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.166+0000 7efd55ffb640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd440744b0 0x7efd440769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.166+0000 7efd55ffb640 1 -- 172.21.15.28:0/60051944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6187+0+0 (secure 0 0 0) 0x7efd5c095650 con 0x7efd600b7660 2024-04-25T19:13:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.166+0000 7efd64e74640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd440744b0 0x7efd440769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.166+0000 7efd64e74640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd440744b0 0x7efd440769c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efd50034000 tx=0x7efd5002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.170+0000 7efd55ffb640 1 -- 172.21.15.28:0/60051944 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd5c05fc00 con 0x7efd600b7660 2024-04-25T19:13:01.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.357+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd60050740 con 0x7efd600b7660 2024-04-25T19:13:01.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.358+0000 7efd55ffb640 1 -- 172.21.15.28:0/60051944 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17009 (secure 0 0 0) 0x7efd5c05f500 con 0x7efd600b7660 2024-04-25T19:13:01.360 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:01.360 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":42,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:12:55.743611+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd440744b0 msgr2=0x7efd440769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd440744b0 0x7efd440769c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7efd50034000 tx=0x7efd5002e000 comp rx=0 tx=0).stop 2024-04-25T19:13:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 msgr2=0x7efd6015b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd6015b8d0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7efd5c00e840 tx=0x7efd5c00ed00 comp rx=0 tx=0).stop 2024-04-25T19:13:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 shutdown_connections 2024-04-25T19:13:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efd440744b0 0x7efd440769c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efd60164990 0x7efd600b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efd601606f0 0x7efd600b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 --2- 172.21.15.28:0/60051944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd600b7660 0x7efd6015b8d0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 >> 172.21.15.28:0/60051944 conn(0x7efd600d2840 msgr2=0x7efd60160b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 shutdown_connections 2024-04-25T19:13:01.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.360+0000 7efd678d9640 1 -- 172.21.15.28:0/60051944 wait complete. 2024-04-25T19:13:01.374 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:12:41.809696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:12:43.014197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:13:01.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:13:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.486+0000 7f4f845b2640 1 Processor -- start 2024-04-25T19:13:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.487+0000 7f4f845b2640 1 -- start start 2024-04-25T19:13:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f845b2640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4f7c0b32d0 0x7f4f7c0b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f845b2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c0b3c20 0x7f4f7c0d1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f845b2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c09d1a0 con 0x7f4f7c09cd90 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f845b2640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f4f7c06df50 con 0x7f4f7c0b32d0 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f845b2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c0d4870 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f81b4d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4f7c0b32d0 0x7f4f7c0b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f82b4f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c0b3c20 0x7f4f7c0d1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f81b4d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4f7c0b32d0 0x7f4f7c0b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39768/0 (socket says 172.21.15.28:39768) 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.488+0000 7f4f81b4d640 1 -- 172.21.15.28:0/242917781 learned_addr learned my addr 172.21.15.28:0/242917781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.489+0000 7f4f81b4d640 1 -- 172.21.15.28:0/242917781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c0b3c20 msgr2=0x7f4f7c0d1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.489+0000 7f4f81b4d640 1 --2- 172.21.15.28:0/242917781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c0b3c20 0x7f4f7c0d1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.489+0000 7f4f81b4d640 1 -- 172.21.15.28:0/242917781 >> v1:172.21.15.53:6789/0 conn(0x7f4f7c09cd90 legacy=0x7f4f7c06e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.489+0000 7f4f81b4d640 1 -- 172.21.15.28:0/242917781 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c06d960 con 0x7f4f7c0b32d0 2024-04-25T19:13:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.490+0000 7f4f81b4d640 1 --2- 172.21.15.28:0/242917781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 0x7f4f7c0b36e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009870 tx=0x7f4f6c02a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=713c1254f8c087f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.490+0000 7f4f8134c640 1 -- 172.21.15.28:0/242917781 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c037070 con 0x7f4f7c0b32d0 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.490+0000 7f4f8134c640 1 -- 172.21.15.28:0/242917781 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f6c02ad60 con 0x7f4f7c0b32d0 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.490+0000 7f4f8134c640 1 -- 172.21.15.28:0/242917781 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c032660 con 0x7f4f7c0b32d0 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.490+0000 7f4f845b2640 1 -- 172.21.15.28:0/242917781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 msgr2=0x7f4f7c0b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.490+0000 7f4f845b2640 1 --2- 172.21.15.28:0/242917781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 0x7f4f7c0b36e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009870 tx=0x7f4f6c02a0d0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.491+0000 7f4f845b2640 1 -- 172.21.15.28:0/242917781 shutdown_connections 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.491+0000 7f4f845b2640 1 --2- 172.21.15.28:0/242917781 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c0b3c20 0x7f4f7c0d1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.491+0000 7f4f845b2640 1 --2- 172.21.15.28:0/242917781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 0x7f4f7c0b36e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.491+0000 7f4f845b2640 1 -- 172.21.15.28:0/242917781 >> 172.21.15.28:0/242917781 conn(0x7f4f7c0d03f0 msgr2=0x7f4f7c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.491+0000 7f4f845b2640 1 -- 172.21.15.28:0/242917781 shutdown_connections 2024-04-25T19:13:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.491+0000 7f4f845b2640 1 -- 172.21.15.28:0/242917781 wait complete. 2024-04-25T19:13:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.492+0000 7f4f845b2640 1 Processor -- start 2024-04-25T19:13:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.492+0000 7f4f845b2640 1 -- start start 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f845b2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c09cd90 0x7f4f7c0b1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f845b2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 0x7f4f7c0b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f845b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f7c0b3c20 0x7f4f7c0cb190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f845b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c0b9180 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f845b2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c06dc70 con 0x7f4f7c0b32d0 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f845b2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c06ea10 con 0x7f4f7c09cd90 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f82b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f7c0b3c20 0x7f4f7c0cb190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f82b4f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f7c0b3c20 0x7f4f7c0cb190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51060/0 (socket says 172.21.15.28:51060) 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f82b4f640 1 -- 172.21.15.28:0/3057591944 learned_addr learned my addr 172.21.15.28:0/3057591944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f8234e640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c09cd90 0x7f4f7c0b1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f81b4d640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 0x7f4f7c0b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f82b4f640 1 -- 172.21.15.28:0/3057591944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c09cd90 msgr2=0x7f4f7c0b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f82b4f640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c09cd90 0x7f4f7c0b1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f82b4f640 1 -- 172.21.15.28:0/3057591944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 msgr2=0x7f4f7c0b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f82b4f640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 0x7f4f7c0b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.494+0000 7f4f82b4f640 1 -- 172.21.15.28:0/3057591944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c0cf910 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.495+0000 7f4f82b4f640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f7c0b3c20 0x7f4f7c0cb190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800efc0 tx=0x7f4f7800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.495+0000 7f4f72ffd640 1 -- 172.21.15.28:0/3057591944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78018020 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.495+0000 7f4f72ffd640 1 -- 172.21.15.28:0/3057591944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f78014480 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.495+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7c0ce1f0 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.495+0000 7f4f72ffd640 1 -- 172.21.15.28:0/3057591944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78009e60 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.495+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f7c0cbdf0 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.496+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f7c0cf680 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.497+0000 7f4f72ffd640 1 -- 172.21.15.28:0/3057591944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4f78005d10 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.497+0000 7f4f72ffd640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4f600744b0 0x7f4f600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.497+0000 7f4f8234e640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4f600744b0 0x7f4f600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.497+0000 7f4f72ffd640 1 -- 172.21.15.28:0/3057591944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6187+0+0 (secure 0 0 0) 0x7f4f78096050 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.498+0000 7f4f8234e640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4f600744b0 0x7f4f600769c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c075dc0 tx=0x7f4f74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.502+0000 7f4f72ffd640 1 -- 172.21.15.28:0/3057591944 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f78022030 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.688+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f7c050740 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.688+0000 7f4f72ffd640 1 -- 172.21.15.28:0/3057591944 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f4f78010040 con 0x7f4f7c0b3c20 2024-04-25T19:13:01.690 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:13:01.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4f600744b0 msgr2=0x7f4f600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4f600744b0 0x7f4f600769c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c075dc0 tx=0x7f4f74009040 comp rx=0 tx=0).stop 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f7c0b3c20 msgr2=0x7f4f7c0cb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f7c0b3c20 0x7f4f7c0cb190 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800efc0 tx=0x7f4f7800c490 comp rx=0 tx=0).stop 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 shutdown_connections 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4f600744b0 0x7f4f600769c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f7c0b3c20 0x7f4f7c0cb190 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f7c0b32d0 0x7f4f7c0b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 --2- 172.21.15.28:0/3057591944 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4f7c09cd90 0x7f4f7c0b1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 >> 172.21.15.28:0/3057591944 conn(0x7f4f7c0d03f0 msgr2=0x7f4f7c0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 shutdown_connections 2024-04-25T19:13:01.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.690+0000 7f4f845b2640 1 -- 172.21.15.28:0/3057591944 wait complete. 2024-04-25T19:13:01.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:13:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.816+0000 7f19638af640 1 Processor -- start 2024-04-25T19:13:01.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.816+0000 7f19638af640 1 -- start start 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.817+0000 7f19638af640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f195c160180 0x7f195c163c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.817+0000 7f19638af640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d2550 0x7f195c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.817+0000 7f19638af640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f195c164bb0 con 0x7f195c0d06c0 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.817+0000 7f19638af640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f195c0d4510 con 0x7f195c160180 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.817+0000 7f19638af640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f195c0d2ea0 con 0x7f195c0d2550 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.817+0000 7f1960e4a640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f195c0d06c0 0x7f195c0b9090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40908/0 (socket says 172.21.15.28:40908) 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.817+0000 7f1960e4a640 1 -- 172.21.15.28:0/3301715549 learned_addr learned my addr 172.21.15.28:0/3301715549 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.817+0000 7f196164b640 1 --2- 172.21.15.28:0/3301715549 >> v2:172.21.15.53:3300/0 conn(0x7f195c160180 0x7f195c163c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.818+0000 7f1961e4c640 1 --2- 172.21.15.28:0/3301715549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d2550 0x7f195c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.818+0000 7f1953fff640 1 -- 172.21.15.28:0/3301715549 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3962857475 0 0) 0x7f195c164bb0 con 0x7f195c0d06c0 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.818+0000 7f1953fff640 1 -- 172.21.15.28:0/3301715549 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1948003520 con 0x7f195c0d06c0 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.818+0000 7f196164b640 1 -- 172.21.15.28:0/3301715549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d2550 msgr2=0x7f195c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.818+0000 7f196164b640 1 --2- 172.21.15.28:0/3301715549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d2550 0x7f195c0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.818+0000 7f196164b640 1 -- 172.21.15.28:0/3301715549 >> v1:172.21.15.28:6789/0 conn(0x7f195c0d06c0 legacy=0x7f195c0b9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.818+0000 7f196164b640 1 -- 172.21.15.28:0/3301715549 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c164310 con 0x7f195c160180 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.819+0000 7f196164b640 1 --2- 172.21.15.28:0/3301715549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c160180 0x7f195c163c80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f195c0d4690 tx=0x7f195802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2343ad6dd381d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.819+0000 7f1953fff640 1 -- 172.21.15.28:0/3301715549 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1958037070 con 0x7f195c160180 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.819+0000 7f1953fff640 1 -- 172.21.15.28:0/3301715549 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1958032440 con 0x7f195c160180 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.819+0000 7f1953fff640 1 -- 172.21.15.28:0/3301715549 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19580327f0 con 0x7f195c160180 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.820+0000 7f19638af640 1 -- 172.21.15.28:0/3301715549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c160180 msgr2=0x7f195c163c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.820+0000 7f19638af640 1 --2- 172.21.15.28:0/3301715549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c160180 0x7f195c163c80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f195c0d4690 tx=0x7f195802a180 comp rx=0 tx=0).stop 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.820+0000 7f19638af640 1 -- 172.21.15.28:0/3301715549 shutdown_connections 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.820+0000 7f19638af640 1 --2- 172.21.15.28:0/3301715549 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d2550 0x7f195c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.820+0000 7f19638af640 1 --2- 172.21.15.28:0/3301715549 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c160180 0x7f195c163c80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.820+0000 7f19638af640 1 -- 172.21.15.28:0/3301715549 >> 172.21.15.28:0/3301715549 conn(0x7f195c06c750 msgr2=0x7f195c06cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.820+0000 7f19638af640 1 -- 172.21.15.28:0/3301715549 shutdown_connections 2024-04-25T19:13:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.820+0000 7f19638af640 1 -- 172.21.15.28:0/3301715549 wait complete. 2024-04-25T19:13:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.822+0000 7f19638af640 1 Processor -- start 2024-04-25T19:13:01.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.822+0000 7f19638af640 1 -- start start 2024-04-25T19:13:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f19638af640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d06c0 0x7f195c15a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f19638af640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c0d2550 0x7f195c0cee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f19638af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195c160180 0x7f195c0b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f19638af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f195c0d0ad0 con 0x7f195c160180 2024-04-25T19:13:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f19638af640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f195c1648d0 con 0x7f195c0d2550 2024-04-25T19:13:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f19638af640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f195c0d49a0 con 0x7f195c0d06c0 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1961e4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195c160180 0x7f195c0b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1961e4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195c160180 0x7f195c0b7830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51076/0 (socket says 172.21.15.28:51076) 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1961e4c640 1 -- 172.21.15.28:0/4100101971 learned_addr learned my addr 172.21.15.28:0/4100101971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1960e4a640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c0d2550 0x7f195c0cee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f196164b640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d06c0 0x7f195c15a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1961e4c640 1 -- 172.21.15.28:0/4100101971 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d06c0 msgr2=0x7f195c15a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1961e4c640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d06c0 0x7f195c15a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1961e4c640 1 -- 172.21.15.28:0/4100101971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c0d2550 msgr2=0x7f195c0cee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1961e4c640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c0d2550 0x7f195c0cee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.823+0000 7f1961e4c640 1 -- 172.21.15.28:0/4100101971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c15a300 con 0x7f195c160180 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.824+0000 7f1961e4c640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195c160180 0x7f195c0b7830 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1954009cb0 tx=0x7f195400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.824+0000 7f1951ffb640 1 -- 172.21.15.28:0/4100101971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954018020 con 0x7f195c160180 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.824+0000 7f1951ffb640 1 -- 172.21.15.28:0/4100101971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1954014480 con 0x7f195c160180 2024-04-25T19:13:01.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.824+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195c158500 con 0x7f195c160180 2024-04-25T19:13:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.824+0000 7f1951ffb640 1 -- 172.21.15.28:0/4100101971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1954014870 con 0x7f195c160180 2024-04-25T19:13:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.824+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195c158a80 con 0x7f195c160180 2024-04-25T19:13:01.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.825+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f195c15ae50 con 0x7f195c160180 2024-04-25T19:13:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.826+0000 7f1951ffb640 1 -- 172.21.15.28:0/4100101971 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1954005d10 con 0x7f195c160180 2024-04-25T19:13:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.826+0000 7f1951ffb640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19400744b0 0x7f19400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.826+0000 7f1951ffb640 1 -- 172.21.15.28:0/4100101971 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6187+0+0 (secure 0 0 0) 0x7f1954096140 con 0x7f195c160180 2024-04-25T19:13:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.826+0000 7f196164b640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19400744b0 0x7f19400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:01.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.827+0000 7f196164b640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19400744b0 0x7f19400769c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1958034000 tx=0x7f195802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:01.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:01.831+0000 7f1951ffb640 1 -- 172.21.15.28:0/4100101971 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1954022030 con 0x7f195c160180 2024-04-25T19:13:02.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.018+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f195c0d3f90 con 0x7f195c160180 2024-04-25T19:13:02.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.019+0000 7f1951ffb640 1 -- 172.21.15.28:0/4100101971 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7f1954010040 con 0x7f195c160180 2024-04-25T19:13:02.021 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19400744b0 msgr2=0x7f19400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19400744b0 0x7f19400769c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1958034000 tx=0x7f195802e000 comp rx=0 tx=0).stop 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195c160180 msgr2=0x7f195c0b7830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195c160180 0x7f195c0b7830 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f1954009cb0 tx=0x7f195400c490 comp rx=0 tx=0).stop 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 shutdown_connections 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f19400744b0 0x7f19400769c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f195c160180 0x7f195c0b7830 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f195c0d2550 0x7f195c0cee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 --2- 172.21.15.28:0/4100101971 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f195c0d06c0 0x7f195c15a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 >> 172.21.15.28:0/4100101971 conn(0x7f195c06c750 msgr2=0x7f195c0d5c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:02.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 shutdown_connections 2024-04-25T19:13:02.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.021+0000 7f19638af640 1 -- 172.21.15.28:0/4100101971 wait complete. 2024-04-25T19:13:02.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:13:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.146+0000 7f8a73869640 1 Processor -- start 2024-04-25T19:13:02.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.146+0000 7f8a73869640 1 -- start start 2024-04-25T19:13:02.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.147+0000 7f8a73869640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8a6c160310 0x7f8a6c0d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.147+0000 7f8a73869640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a6c0b79f0 0x7f8a6c0b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a73869640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a6c160720 con 0x7f8a6c08cce0 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a73869640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8a6c0b8340 con 0x7f8a6c0b79f0 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a73869640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f8a6c0d5640 con 0x7f8a6c160310 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a70e04640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a6c0b79f0 0x7f8a6c0b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a70e04640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a6c0b79f0 0x7f8a6c0b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51092/0 (socket says 172.21.15.28:51092) 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a70e04640 1 -- 172.21.15.28:0/2632047095 learned_addr learned my addr 172.21.15.28:0/2632047095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a71605640 1 --2- 172.21.15.28:0/2632047095 >> v2:172.21.15.53:3300/0 conn(0x7f8a6c160310 0x7f8a6c0d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a70e04640 1 -- 172.21.15.28:0/2632047095 >> v2:172.21.15.53:3300/0 conn(0x7f8a6c160310 msgr2=0x7f8a6c0d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a70e04640 1 --2- 172.21.15.28:0/2632047095 >> v2:172.21.15.53:3300/0 conn(0x7f8a6c160310 0x7f8a6c0d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a70e04640 1 -- 172.21.15.28:0/2632047095 >> v1:172.21.15.53:6789/0 conn(0x7f8a6c08cce0 legacy=0x7f8a6c08d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a70e04640 1 -- 172.21.15.28:0/2632047095 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c0d0f40 con 0x7f8a6c0b79f0 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.148+0000 7f8a71605640 1 --2- 172.21.15.28:0/2632047095 >> v2:172.21.15.53:3300/0 conn(0x7f8a6c160310 0x7f8a6c0d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:02.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.149+0000 7f8a70e04640 1 --2- 172.21.15.28:0/2632047095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c0b79f0 0x7f8a6c0b7e00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0d6be0 tx=0x7f8a6002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6274d9e243ae82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.149+0000 7f8a6bfff640 1 -- 172.21.15.28:0/2632047095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60037070 con 0x7f8a6c0b79f0 2024-04-25T19:13:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.149+0000 7f8a6bfff640 1 -- 172.21.15.28:0/2632047095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a60032440 con 0x7f8a6c0b79f0 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.149+0000 7f8a6bfff640 1 -- 172.21.15.28:0/2632047095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a60032810 con 0x7f8a6c0b79f0 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.150+0000 7f8a73869640 1 -- 172.21.15.28:0/2632047095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c0b79f0 msgr2=0x7f8a6c0b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.150+0000 7f8a73869640 1 --2- 172.21.15.28:0/2632047095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c0b79f0 0x7f8a6c0b7e00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0d6be0 tx=0x7f8a6002a200 comp rx=0 tx=0).stop 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.150+0000 7f8a73869640 1 -- 172.21.15.28:0/2632047095 shutdown_connections 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.150+0000 7f8a73869640 1 --2- 172.21.15.28:0/2632047095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c0b79f0 0x7f8a6c0b7e00 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.150+0000 7f8a73869640 1 --2- 172.21.15.28:0/2632047095 >> v2:172.21.15.53:3300/0 conn(0x7f8a6c160310 0x7f8a6c0d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.150+0000 7f8a73869640 1 -- 172.21.15.28:0/2632047095 >> 172.21.15.28:0/2632047095 conn(0x7f8a6c0d2a50 msgr2=0x7f8a6c0d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.150+0000 7f8a73869640 1 -- 172.21.15.28:0/2632047095 shutdown_connections 2024-04-25T19:13:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.150+0000 7f8a73869640 1 -- 172.21.15.28:0/2632047095 wait complete. 2024-04-25T19:13:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.152+0000 7f8a73869640 1 Processor -- start 2024-04-25T19:13:02.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.152+0000 7f8a73869640 1 -- start start 2024-04-25T19:13:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a73869640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c08cce0 0x7f8a6c1e7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a73869640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a6c0b79f0 0x7f8a6c0cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a73869640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a6c160310 0x7f8a6c0cc240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a73869640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c0b9400 con 0x7f8a6c08cce0 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a73869640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6c0d1500 con 0x7f8a6c160310 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a73869640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8a6c0d0bf0 con 0x7f8a6c0b79f0 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a71605640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c08cce0 0x7f8a6c1e7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a71605640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c08cce0 0x7f8a6c1e7140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51100/0 (socket says 172.21.15.28:51100) 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a71605640 1 -- 172.21.15.28:0/440521176 learned_addr learned my addr 172.21.15.28:0/440521176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.153+0000 7f8a71e06640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a6c160310 0x7f8a6c0cc240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a70e04640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a6c0b79f0 0x7f8a6c0cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a71605640 1 -- 172.21.15.28:0/440521176 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a6c0b79f0 msgr2=0x7f8a6c0cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a71605640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a6c0b79f0 0x7f8a6c0cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a71605640 1 -- 172.21.15.28:0/440521176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a6c160310 msgr2=0x7f8a6c0cc240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a71605640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a6c160310 0x7f8a6c0cc240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a71605640 1 -- 172.21.15.28:0/440521176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6c0c9860 con 0x7f8a6c08cce0 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a71605640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c08cce0 0x7f8a6c1e7140 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400d7d0 tx=0x7f8a6400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a69ffb640 1 -- 172.21.15.28:0/440521176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64004030 con 0x7f8a6c08cce0 2024-04-25T19:13:02.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a69ffb640 1 -- 172.21.15.28:0/440521176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a640041d0 con 0x7f8a6c08cce0 2024-04-25T19:13:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6c0c9b40 con 0x7f8a6c08cce0 2024-04-25T19:13:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a69ffb640 1 -- 172.21.15.28:0/440521176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a640096a0 con 0x7f8a6c08cce0 2024-04-25T19:13:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.154+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a6c1f76b0 con 0x7f8a6c08cce0 2024-04-25T19:13:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.156+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a6c0cfee0 con 0x7f8a6c08cce0 2024-04-25T19:13:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.156+0000 7f8a69ffb640 1 -- 172.21.15.28:0/440521176 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8a64009840 con 0x7f8a6c08cce0 2024-04-25T19:13:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.156+0000 7f8a69ffb640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a580744b0 0x7f8a580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.157+0000 7f8a70e04640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a580744b0 0x7f8a580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.157+0000 7f8a69ffb640 1 -- 172.21.15.28:0/440521176 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6187+0+0 (secure 0 0 0) 0x7f8a64095720 con 0x7f8a6c08cce0 2024-04-25T19:13:02.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.157+0000 7f8a70e04640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a580744b0 0x7f8a580769c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a60034000 tx=0x7f8a6002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:02.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.161+0000 7f8a69ffb640 1 -- 172.21.15.28:0/440521176 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a6405fcd0 con 0x7f8a6c08cce0 2024-04-25T19:13:02.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:13:02.233 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:13:02.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-04-25T19:13:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.346+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a6c0d20a0 con 0x7f8a6c08cce0 2024-04-25T19:13:02.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.346+0000 7f8a69ffb640 1 -- 172.21.15.28:0/440521176 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f8a6405f5d0 con 0x7f8a6c08cce0 2024-04-25T19:13:02.350 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:13:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a580744b0 msgr2=0x7f8a580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a580744b0 0x7f8a580769c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a60034000 tx=0x7f8a6002e000 comp rx=0 tx=0).stop 2024-04-25T19:13:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c08cce0 msgr2=0x7f8a6c1e7140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c08cce0 0x7f8a6c1e7140 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400d7d0 tx=0x7f8a6400dc90 comp rx=0 tx=0).stop 2024-04-25T19:13:02.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 shutdown_connections 2024-04-25T19:13:02.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a580744b0 0x7f8a580769c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a6c160310 0x7f8a6c0cc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a6c0b79f0 0x7f8a6c0cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 --2- 172.21.15.28:0/440521176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a6c08cce0 0x7f8a6c1e7140 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 >> 172.21.15.28:0/440521176 conn(0x7f8a6c0d2a50 msgr2=0x7f8a6c15c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:02.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 shutdown_connections 2024-04-25T19:13:02.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.348+0000 7f8a73869640 1 -- 172.21.15.28:0/440521176 wait complete. 2024-04-25T19:13:02.361 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'recovery_fs' 2024-04-25T19:13:02.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create recovery_fs_metadata --pg_num_min 64 2024-04-25T19:13:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.475+0000 7fba0d996640 1 Processor -- start 2024-04-25T19:13:02.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.475+0000 7fba0d996640 1 -- start start 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba0d996640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba080ce980 0x7fba080ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba0d996640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fba080d1e40 0x7fba080d2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba0d996640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba080cdd10 con 0x7fba080b9210 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba0d996640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fba080d37a0 con 0x7fba080d1e40 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba0d996640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fba0809cf80 con 0x7fba080ce980 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba06ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fba080b9210 0x7fba0806e490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40912/0 (socket says 172.21.15.28:40912) 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba06ffd640 1 -- 172.21.15.28:0/1025273962 learned_addr learned my addr 172.21.15.28:0/1025273962 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba077fe640 1 --2- 172.21.15.28:0/1025273962 >> v2:172.21.15.53:3300/0 conn(0x7fba080d1e40 0x7fba080d2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba067fc640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba080ce980 0x7fba080ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626399310 0 0) 0x7fba080cdd10 con 0x7fba080b9210 2024-04-25T19:13:02.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb9f0003520 con 0x7fba080b9210 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1422167766 0 0) 0x7fb9f0003520 con 0x7fba080b9210 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba080ce980 msgr2=0x7fba080ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba080ce980 0x7fba080ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 >> v2:172.21.15.53:3300/0 conn(0x7fba080d1e40 msgr2=0x7fba080d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 --2- 172.21.15.28:0/1025273962 >> v2:172.21.15.53:3300/0 conn(0x7fba080d1e40 0x7fba080d2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba0806de00 con 0x7fba080b9210 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fba000017f0 con 0x7fba080b9210 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 >> v1:172.21.15.28:6789/0 conn(0x7fba080b9210 legacy=0x7fba0806e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00039b0 0x7fb9f0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.477+0000 7fba05ffb640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba080ce980 0x7fb9f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba05ffb640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9f0004d40 0x7fb9f0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba080d37a0 con 0x7fb9f00039b0 2024-04-25T19:13:02.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f000b9c0 con 0x7fb9f0004d40 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f000bbc0 con 0x7fba080ce980 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba06ffd640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00039b0 0x7fb9f0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba067fc640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba080ce980 0x7fb9f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba077fe640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9f0004d40 0x7fb9f0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba06ffd640 1 -- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba080ce980 msgr2=0x7fb9f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba06ffd640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fba080ce980 0x7fb9f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba06ffd640 1 -- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9f0004d40 msgr2=0x7fb9f0005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba06ffd640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9f0004d40 0x7fb9f0005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba06ffd640 1 -- 172.21.15.28:0/1025273962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f0003520 con 0x7fb9f00039b0 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.478+0000 7fba077fe640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9f0004d40 0x7fb9f0005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.479+0000 7fba06ffd640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00039b0 0x7fb9f0003e20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fba0000c240 tx=0x7fb9f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6712a76a5aeeee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.479+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba00007d40 con 0x7fb9f00039b0 2024-04-25T19:13:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.479+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba00007290 con 0x7fb9f00039b0 2024-04-25T19:13:02.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.479+0000 7fba05ffb640 1 -- 172.21.15.28:0/1025273962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba000057a0 con 0x7fb9f00039b0 2024-04-25T19:13:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.479+0000 7fba0d996640 1 -- 172.21.15.28:0/1025273962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00039b0 msgr2=0x7fb9f0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.479+0000 7fba0d996640 1 --2- 172.21.15.28:0/1025273962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb9f00039b0 0x7fb9f0003e20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fba0000c240 tx=0x7fb9f0005830 comp rx=0 tx=0).stop 2024-04-25T19:13:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.480+0000 7fba077fe640 1 -- 172.21.15.28:0/1025273962 reap_dead start 2024-04-25T19:13:02.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.480+0000 7fba0d996640 1 -- 172.21.15.28:0/1025273962 shutdown_connections 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.480+0000 7fba0d996640 1 -- 172.21.15.28:0/1025273962 >> 172.21.15.28:0/1025273962 conn(0x7fba080d03a0 msgr2=0x7fba080d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.480+0000 7fba0d996640 1 -- 172.21.15.28:0/1025273962 shutdown_connections 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.480+0000 7fba0d996640 1 -- 172.21.15.28:0/1025273962 wait complete. 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.481+0000 7fba0d996640 1 Processor -- start 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.482+0000 7fba0d996640 1 -- start start 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba0d996640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb9f00039b0 0x7fba081f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba0d996640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba080cc7f0 0x7fba080ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba0d996640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba080c8980 0x7fba080c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba0d996640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fba080d0ec0 con 0x7fba080c8980 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba0d996640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fba0806eba0 con 0x7fba080cc7f0 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba0d996640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fba080d1040 con 0x7fb9f00039b0 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba080c8980 0x7fba080c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba077fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba080c8980 0x7fba080c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51116/0 (socket says 172.21.15.28:51116) 2024-04-25T19:13:02.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba077fe640 1 -- 172.21.15.28:0/275677400 learned_addr learned my addr 172.21.15.28:0/275677400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba06ffd640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb9f00039b0 0x7fba081f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba067fc640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba080cc7f0 0x7fba080ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba077fe640 1 -- 172.21.15.28:0/275677400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb9f00039b0 msgr2=0x7fba081f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba077fe640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb9f00039b0 0x7fba081f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba077fe640 1 -- 172.21.15.28:0/275677400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba080cc7f0 msgr2=0x7fba080ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba077fe640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba080cc7f0 0x7fba080ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.483+0000 7fba077fe640 1 -- 172.21.15.28:0/275677400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba080cd140 con 0x7fba080c8980 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.484+0000 7fba077fe640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba080c8980 0x7fba080c8d90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba080694f0 tx=0x7fb9f80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.484+0000 7fb9effff640 1 -- 172.21.15.28:0/275677400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f801a020 con 0x7fba080c8980 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.484+0000 7fb9effff640 1 -- 172.21.15.28:0/275677400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f8011e10 con 0x7fba080c8980 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.484+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba080c7bc0 con 0x7fba080c8980 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.484+0000 7fb9effff640 1 -- 172.21.15.28:0/275677400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f8007e60 con 0x7fba080c8980 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.484+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba080c8180 con 0x7fba080c8980 2024-04-25T19:13:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.485+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba080b7f40 con 0x7fba080c8980 2024-04-25T19:13:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.485+0000 7fb9effff640 1 -- 172.21.15.28:0/275677400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb9f8005d10 con 0x7fba080c8980 2024-04-25T19:13:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.486+0000 7fb9effff640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb9f40744e0 0x7fb9f40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.486+0000 7fb9effff640 1 -- 172.21.15.28:0/275677400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6187+0+0 (secure 0 0 0) 0x7fb9f8095b20 con 0x7fba080c8980 2024-04-25T19:13:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.486+0000 7fba06ffd640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb9f40744e0 0x7fb9f40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:02.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.486+0000 7fba06ffd640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb9f40744e0 0x7fb9f40769f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fba00033000 tx=0x7fba00009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:02.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.490+0000 7fb9effff640 1 -- 172.21.15.28:0/275677400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9f8024070 con 0x7fba080c8980 2024-04-25T19:13:02.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.675+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "recovery_fs_metadata", "pg_num_min": 64} v 0) -- 0x7fba08072a80 con 0x7fba080c8980 2024-04-25T19:13:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.919+0000 7fb9effff640 1 -- 172.21.15.28:0/275677400 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "recovery_fs_metadata", "pg_num_min": 64}]=0 pool 'recovery_fs_metadata' created v44) ==== 148+0+0 (secure 0 0 0) 0x7fba08072a80 con 0x7fba080c8980 2024-04-25T19:13:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'recovery_fs_metadata' created 2024-04-25T19:13:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb9f40744e0 msgr2=0x7fb9f40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb9f40744e0 0x7fb9f40769f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fba00033000 tx=0x7fba00009030 comp rx=0 tx=0).stop 2024-04-25T19:13:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba080c8980 msgr2=0x7fba080c8d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba080c8980 0x7fba080c8d90 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fba080694f0 tx=0x7fb9f80115a0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 shutdown_connections 2024-04-25T19:13:02.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb9f40744e0 0x7fb9f40769f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba080c8980 0x7fba080c8d90 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba080cc7f0 0x7fba080ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 --2- 172.21.15.28:0/275677400 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb9f00039b0 0x7fba081f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:02.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.923+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 >> 172.21.15.28:0/275677400 conn(0x7fba080d03a0 msgr2=0x7fba0806e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:02.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.924+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 shutdown_connections 2024-04-25T19:13:02.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:02.924+0000 7fba0d996640 1 -- 172.21.15.28:0/275677400 wait complete. 2024-04-25T19:13:02.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T19:13:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.058+0000 7f2799e42640 1 Processor -- start 2024-04-25T19:13:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.058+0000 7f2799e42640 1 -- start start 2024-04-25T19:13:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f2799e42640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27940b5f30 0x7f27940b8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f2799e42640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27940ce8c0 0x7f27940cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f2799e42640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f279409cea0 con 0x7f27940d2cc0 2024-04-25T19:13:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f2799e42640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f279406eca0 con 0x7f27940b5f30 2024-04-25T19:13:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f2799e42640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f279406e9c0 con 0x7f27940ce8c0 2024-04-25T19:13:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f27937fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27940b5f30 0x7f27940b8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f27937fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f27940b5f30 0x7f27940b8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51120/0 (socket says 172.21.15.28:51120) 2024-04-25T19:13:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f27937fe640 1 -- 172.21.15.28:0/1653512614 learned_addr learned my addr 172.21.15.28:0/1653512614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f2792ffd640 1 --1- 172.21.15.28:0/1653512614 >> v1:172.21.15.28:6789/0 conn(0x7f27940d2cc0 0x7f27940d2090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40922/0 (socket says 172.21.15.28:40922) 2024-04-25T19:13:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f2793fff640 1 --2- 172.21.15.28:0/1653512614 >> v2:172.21.15.53:3300/0 conn(0x7f27940ce8c0 0x7f27940cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f27937fe640 1 -- 172.21.15.28:0/1653512614 >> v2:172.21.15.53:3300/0 conn(0x7f27940ce8c0 msgr2=0x7f27940cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f27937fe640 1 --2- 172.21.15.28:0/1653512614 >> v2:172.21.15.53:3300/0 conn(0x7f27940ce8c0 0x7f27940cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f27937fe640 1 -- 172.21.15.28:0/1653512614 >> v1:172.21.15.28:6789/0 conn(0x7f27940d2cc0 legacy=0x7f27940d2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f27937fe640 1 -- 172.21.15.28:0/1653512614 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279406e690 con 0x7f27940b5f30 2024-04-25T19:13:03.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.060+0000 7f27927fc640 1 -- 172.21.15.28:0/1653512614 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1495256902 0 0) 0x7f279409cea0 con 0x7f27940d2cc0 2024-04-25T19:13:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.061+0000 7f27937fe640 1 --2- 172.21.15.28:0/1653512614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f27940b8c20 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2794071470 tx=0x7f278402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d94f84f085753d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.061+0000 7f27927fc640 1 -- 172.21.15.28:0/1653512614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784004030 con 0x7f27940b5f30 2024-04-25T19:13:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.061+0000 7f27927fc640 1 -- 172.21.15.28:0/1653512614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27840041d0 con 0x7f27940b5f30 2024-04-25T19:13:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.061+0000 7f27927fc640 1 -- 172.21.15.28:0/1653512614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784003610 con 0x7f27940b5f30 2024-04-25T19:13:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.062+0000 7f2799e42640 1 -- 172.21.15.28:0/1653512614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 msgr2=0x7f27940b8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.062+0000 7f2799e42640 1 --2- 172.21.15.28:0/1653512614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f27940b8c20 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2794071470 tx=0x7f278402a1c0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.062+0000 7f2799e42640 1 -- 172.21.15.28:0/1653512614 shutdown_connections 2024-04-25T19:13:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.062+0000 7f2799e42640 1 --2- 172.21.15.28:0/1653512614 >> v2:172.21.15.53:3300/0 conn(0x7f27940ce8c0 0x7f27940cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.062+0000 7f2799e42640 1 --2- 172.21.15.28:0/1653512614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f27940b8c20 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.062+0000 7f2799e42640 1 -- 172.21.15.28:0/1653512614 >> 172.21.15.28:0/1653512614 conn(0x7f27940b5410 msgr2=0x7f27940b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.062+0000 7f2799e42640 1 -- 172.21.15.28:0/1653512614 shutdown_connections 2024-04-25T19:13:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.062+0000 7f2799e42640 1 -- 172.21.15.28:0/1653512614 wait complete. 2024-04-25T19:13:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.064+0000 7f2799e42640 1 Processor -- start 2024-04-25T19:13:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.064+0000 7f2799e42640 1 -- start start 2024-04-25T19:13:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f2799e42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f279415e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f2799e42640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27940ce8c0 0x7f279415ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f2799e42640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27940d2cc0 0x7f27940b6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f2799e42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27940b9380 con 0x7f27940b5f30 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f2799e42640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f27940d27a0 con 0x7f27940d2cc0 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f2799e42640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f27940d30d0 con 0x7f27940ce8c0 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f27937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f279415e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f27937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f279415e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51126/0 (socket says 172.21.15.28:51126) 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.065+0000 7f27937fe640 1 -- 172.21.15.28:0/2407204138 learned_addr learned my addr 172.21.15.28:0/2407204138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f2793fff640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27940d2cc0 0x7f27940b6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f2792ffd640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27940ce8c0 0x7f279415ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f27937fe640 1 -- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27940ce8c0 msgr2=0x7f279415ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f27937fe640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27940ce8c0 0x7f279415ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f27937fe640 1 -- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27940d2cc0 msgr2=0x7f27940b6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f27937fe640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27940d2cc0 0x7f27940b6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f27937fe640 1 -- 172.21.15.28:0/2407204138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f279415e810 con 0x7f27940b5f30 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f2792ffd640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27940ce8c0 0x7f279415ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f27937fe640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f279415e2d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f278402a200 tx=0x7f27840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f277bfff640 1 -- 172.21.15.28:0/2407204138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784041020 con 0x7f27940b5f30 2024-04-25T19:13:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f277bfff640 1 -- 172.21.15.28:0/2407204138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2784003c50 con 0x7f27940b5f30 2024-04-25T19:13:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f279415ce30 con 0x7f27940b5f30 2024-04-25T19:13:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f277bfff640 1 -- 172.21.15.28:0/2407204138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2784004030 con 0x7f27940b5f30 2024-04-25T19:13:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.066+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f279415c350 con 0x7f27940b5f30 2024-04-25T19:13:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.067+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2794050740 con 0x7f27940b5f30 2024-04-25T19:13:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.068+0000 7f277bfff640 1 -- 172.21.15.28:0/2407204138 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2784044020 con 0x7f27940b5f30 2024-04-25T19:13:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.068+0000 7f277bfff640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27800744e0 0x7f27800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.068+0000 7f277bfff640 1 -- 172.21.15.28:0/2407204138 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6562+0+0 (secure 0 0 0) 0x7f27840b6250 con 0x7f27940b5f30 2024-04-25T19:13:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.068+0000 7f2792ffd640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27800744e0 0x7f27800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.069+0000 7f2792ffd640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27800744e0 0x7f27800769f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f27940b4ce0 tx=0x7f2788009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.073+0000 7f277bfff640 1 -- 172.21.15.28:0/2407204138 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f278404b030 con 0x7f27940b5f30 2024-04-25T19:13:03.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.259+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f27940cf9a0 con 0x7f27940b5f30 2024-04-25T19:13:03.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.260+0000 7f277bfff640 1 -- 172.21.15.28:0/2407204138 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' already exists v44) ==== 177+0+0 (secure 0 0 0) 0x7f2784080680 con 0x7f27940b5f30 2024-04-25T19:13:03.261 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' already exists 2024-04-25T19:13:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27800744e0 msgr2=0x7f27800769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27800744e0 0x7f27800769f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f27940b4ce0 tx=0x7f2788009040 comp rx=0 tx=0).stop 2024-04-25T19:13:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 msgr2=0x7f279415e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f279415e2d0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f278402a200 tx=0x7f27840046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 shutdown_connections 2024-04-25T19:13:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27800744e0 0x7f27800769f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27940d2cc0 0x7f27940b6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27940ce8c0 0x7f279415ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 --2- 172.21.15.28:0/2407204138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27940b5f30 0x7f279415e2d0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 >> 172.21.15.28:0/2407204138 conn(0x7f27940b5410 msgr2=0x7f27940b5820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 shutdown_connections 2024-04-25T19:13:03.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.262+0000 7f2799e42640 1 -- 172.21.15.28:0/2407204138 wait complete. 2024-04-25T19:13:03.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new recovery_fs recovery_fs_metadata cephfs_data --recover --allow-dangerous-metadata-overlay 2024-04-25T19:13:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.390+0000 7f38aae9f640 1 Processor -- start 2024-04-25T19:13:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.390+0000 7f38aae9f640 1 -- start start 2024-04-25T19:13:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.391+0000 7f38aae9f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f38a40b47d0 0x7f38a41601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.391+0000 7f38aae9f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a406e5e0 0x7f38a40b9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.391+0000 7f38aae9f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38a40d12e0 con 0x7f38a40d5690 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.391+0000 7f38aae9f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f38a40d15c0 con 0x7f38a40b47d0 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.391+0000 7f38aae9f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f38a40d4bb0 con 0x7f38a406e5e0 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f38a8c3b640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f38a40b47d0 0x7f38a41601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f38a943c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a406e5e0 0x7f38a40b9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f38a8c3b640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f38a40b47d0 0x7f38a41601f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:39880/0 (socket says 172.21.15.28:39880) 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f38a8c3b640 1 -- 172.21.15.28:0/319935291 learned_addr learned my addr 172.21.15.28:0/319935291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f38a8c3b640 1 -- 172.21.15.28:0/319935291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a406e5e0 msgr2=0x7f38a40b9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f38a8c3b640 1 --2- 172.21.15.28:0/319935291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a406e5e0 0x7f38a40b9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f38a8c3b640 1 -- 172.21.15.28:0/319935291 >> v1:172.21.15.53:6789/0 conn(0x7f38a40d5690 legacy=0x7f38a406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f389b7fe640 1 -- 172.21.15.28:0/319935291 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2758623470 0 0) 0x7f38a40d12e0 con 0x7f38a40d5690 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.392+0000 7f38a8c3b640 1 -- 172.21.15.28:0/319935291 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a40d4e90 con 0x7f38a40b47d0 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.393+0000 7f38a8c3b640 1 --2- 172.21.15.28:0/319935291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a40b47d0 0x7f38a41601f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009870 tx=0x7f38a002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc69bab89120fce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.393+0000 7f389b7fe640 1 -- 172.21.15.28:0/319935291 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0037070 con 0x7f38a40b47d0 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.393+0000 7f389b7fe640 1 -- 172.21.15.28:0/319935291 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a0032440 con 0x7f38a40b47d0 2024-04-25T19:13:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.393+0000 7f389b7fe640 1 -- 172.21.15.28:0/319935291 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0032810 con 0x7f38a40b47d0 2024-04-25T19:13:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.394+0000 7f38aae9f640 1 -- 172.21.15.28:0/319935291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a40b47d0 msgr2=0x7f38a41601f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.394+0000 7f38aae9f640 1 --2- 172.21.15.28:0/319935291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a40b47d0 0x7f38a41601f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f38a0009870 tx=0x7f38a002a270 comp rx=0 tx=0).stop 2024-04-25T19:13:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.394+0000 7f38aae9f640 1 -- 172.21.15.28:0/319935291 shutdown_connections 2024-04-25T19:13:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.394+0000 7f38aae9f640 1 --2- 172.21.15.28:0/319935291 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a406e5e0 0x7f38a40b9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.394+0000 7f38aae9f640 1 --2- 172.21.15.28:0/319935291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a40b47d0 0x7f38a41601f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.394+0000 7f38aae9f640 1 -- 172.21.15.28:0/319935291 >> 172.21.15.28:0/319935291 conn(0x7f38a40d2c60 msgr2=0x7f38a40d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.394+0000 7f38aae9f640 1 -- 172.21.15.28:0/319935291 shutdown_connections 2024-04-25T19:13:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.394+0000 7f38aae9f640 1 -- 172.21.15.28:0/319935291 wait complete. 2024-04-25T19:13:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.396+0000 7f38aae9f640 1 Processor -- start 2024-04-25T19:13:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.396+0000 7f38aae9f640 1 -- start start 2024-04-25T19:13:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38aae9f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a40b47d0 0x7f38a40cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38aae9f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a40d5690 0x7f38a415b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38aae9f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a415bf00 0x7f38a415c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38aae9f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38a40b4d00 con 0x7f38a40b47d0 2024-04-25T19:13:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38aae9f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f38a4160730 con 0x7f38a415bf00 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38aae9f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f38a40b4e80 con 0x7f38a40d5690 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38a8c3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a40b47d0 0x7f38a40cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38a8c3b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a40b47d0 0x7f38a40cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51132/0 (socket says 172.21.15.28:51132) 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38a8c3b640 1 -- 172.21.15.28:0/175156275 learned_addr learned my addr 172.21.15.28:0/175156275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.397+0000 7f38a943c640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a415bf00 0x7f38a415c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f389bfff640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a40d5690 0x7f38a415b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38a8c3b640 1 -- 172.21.15.28:0/175156275 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a40d5690 msgr2=0x7f38a415b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38a8c3b640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a40d5690 0x7f38a415b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38a8c3b640 1 -- 172.21.15.28:0/175156275 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a415bf00 msgr2=0x7f38a415c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38a8c3b640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a415bf00 0x7f38a415c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38a8c3b640 1 -- 172.21.15.28:0/175156275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a40cb0f0 con 0x7f38a40b47d0 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38a8c3b640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a40b47d0 0x7f38a40cf6c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f38a00090b0 tx=0x7f38a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38997fa640 1 -- 172.21.15.28:0/175156275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0037040 con 0x7f38a40b47d0 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38997fa640 1 -- 172.21.15.28:0/175156275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a0004030 con 0x7f38a40b47d0 2024-04-25T19:13:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a40cb3d0 con 0x7f38a40b47d0 2024-04-25T19:13:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38997fa640 1 -- 172.21.15.28:0/175156275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a0004400 con 0x7f38a40b47d0 2024-04-25T19:13:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.398+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a40cb990 con 0x7f38a40b47d0 2024-04-25T19:13:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.399+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a406a100 con 0x7f38a40b47d0 2024-04-25T19:13:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.400+0000 7f38997fa640 1 -- 172.21.15.28:0/175156275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f38a003a070 con 0x7f38a40b47d0 2024-04-25T19:13:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.400+0000 7f38997fa640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f38900744b0 0x7f38900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.400+0000 7f38997fa640 1 -- 172.21.15.28:0/175156275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6562+0+0 (secure 0 0 0) 0x7f38a00b5bc0 con 0x7f38a40b47d0 2024-04-25T19:13:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.400+0000 7f389bfff640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f38900744b0 0x7f38900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.401+0000 7f389bfff640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f38900744b0 0x7f38900769c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f38a415d6d0 tx=0x7f3894009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:03.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.405+0000 7f38997fa640 1 -- 172.21.15.28:0/175156275 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38a0041020 con 0x7f38a40b47d0 2024-04-25T19:13:03.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.629+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "recovery_fs", "metadata": "recovery_fs_metadata", "data": "cephfs_data", "recover": true, "allow_dangerous_metadata_overlay": true} v 0) -- 0x7f38a408e890 con 0x7f38a40b47d0 2024-04-25T19:13:03.921 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-25T19:13:03.920+0000 7fa811aa3640 -1 log_channel(cluster) log [ERR] : Health check update: 2 filesystems are offline (MDS_ALL_DOWN) 2024-04-25T19:13:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.921+0000 7f38997fa640 1 -- 172.21.15.28:0/175156275 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "recovery_fs", "metadata": "recovery_fs_metadata", "data": "cephfs_data", "recover": true, "allow_dangerous_metadata_overlay": true}]=0 new fs with metadata pool 7 and data pool 6 v23) ==== 241+0+0 (secure 0 0 0) 0x7f38a00b5030 con 0x7f38a40b47d0 2024-04-25T19:13:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 7 and data pool 6 2024-04-25T19:13:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.924+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f38900744b0 msgr2=0x7f38900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.924+0000 7f38aae9f640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f38900744b0 0x7f38900769c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f38a415d6d0 tx=0x7f3894009040 comp rx=0 tx=0).stop 2024-04-25T19:13:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.924+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a40b47d0 msgr2=0x7f38a40cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.924+0000 7f38aae9f640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a40b47d0 0x7f38a40cf6c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f38a00090b0 tx=0x7f38a00046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.925+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 shutdown_connections 2024-04-25T19:13:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.925+0000 7f38aae9f640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f38900744b0 0x7f38900769c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.925+0000 7f38aae9f640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38a415bf00 0x7f38a415c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.925+0000 7f38aae9f640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38a40d5690 0x7f38a415b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.925+0000 7f38aae9f640 1 --2- 172.21.15.28:0/175156275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a40b47d0 0x7f38a40cf6c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.925+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 >> 172.21.15.28:0/175156275 conn(0x7f38a40d2c60 msgr2=0x7f38a40b6420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.927+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 shutdown_connections 2024-04-25T19:13:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:03.927+0000 7f38aae9f640 1 -- 172.21.15.28:0/175156275 wait complete. 2024-04-25T19:13:03.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.061+0000 7f8d4c5ee640 1 Processor -- start 2024-04-25T19:13:04.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.061+0000 7f8d4c5ee640 1 -- start start 2024-04-25T19:13:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.062+0000 7f8d4c5ee640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8d4409d010 0x7f8d4406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.062+0000 7f8d4c5ee640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d4406d710 con 0x7f8d4409cb60 2024-04-25T19:13:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.062+0000 7f8d4c5ee640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d4408ce70 con 0x7f8d4406e410 2024-04-25T19:13:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.062+0000 7f8d4c5ee640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f8d4408cc20 con 0x7f8d4409d010 2024-04-25T19:13:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d4ab8b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8d4409cb60 0x7f8d440d1cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40936/0 (socket says 172.21.15.28:40936) 2024-04-25T19:13:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d4ab8b640 1 -- 172.21.15.28:0/1573059807 learned_addr learned my addr 172.21.15.28:0/1573059807 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:04.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d4a38a640 1 --2- 172.21.15.28:0/1573059807 >> v2:172.21.15.53:3300/0 conn(0x7f8d4409d010 0x7f8d4406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d49388640 1 -- 172.21.15.28:0/1573059807 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3681612654 0 0) 0x7f8d4406d710 con 0x7f8d4409cb60 2024-04-25T19:13:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d49388640 1 -- 172.21.15.28:0/1573059807 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d30003520 con 0x7f8d4409cb60 2024-04-25T19:13:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d4a38a640 1 -- 172.21.15.28:0/1573059807 >> v1:172.21.15.53:6789/0 conn(0x7f8d4406e410 legacy=0x7f8d440cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d4a38a640 1 -- 172.21.15.28:0/1573059807 >> v1:172.21.15.28:6789/0 conn(0x7f8d4409cb60 legacy=0x7f8d440d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d49388640 1 -- 172.21.15.28:0/1573059807 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 842368120 0 0) 0x7f8d30003520 con 0x7f8d4409cb60 2024-04-25T19:13:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.063+0000 7f8d4a38a640 1 -- 172.21.15.28:0/1573059807 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d440b92d0 con 0x7f8d4409d010 2024-04-25T19:13:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.064+0000 7f8d4a38a640 1 --2- 172.21.15.28:0/1573059807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 0x7f8d4406ded0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8d4408cda0 tx=0x7f8d4002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90b4bd41d119db0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.064+0000 7f8d49388640 1 -- 172.21.15.28:0/1573059807 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40037070 con 0x7f8d4409d010 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.064+0000 7f8d49388640 1 -- 172.21.15.28:0/1573059807 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4002ac30 con 0x7f8d4409d010 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.064+0000 7f8d49388640 1 -- 172.21.15.28:0/1573059807 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c0019f0 con 0x7f8d4409d010 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.065+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/1573059807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 msgr2=0x7f8d4406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.065+0000 7f8d4c5ee640 1 --2- 172.21.15.28:0/1573059807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 0x7f8d4406ded0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8d4408cda0 tx=0x7f8d4002a1a0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.065+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/1573059807 shutdown_connections 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.065+0000 7f8d4c5ee640 1 --2- 172.21.15.28:0/1573059807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 0x7f8d4406ded0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.065+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/1573059807 >> 172.21.15.28:0/1573059807 conn(0x7f8d440d0430 msgr2=0x7f8d440d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.065+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/1573059807 shutdown_connections 2024-04-25T19:13:04.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.065+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/1573059807 wait complete. 2024-04-25T19:13:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.067+0000 7f8d4c5ee640 1 Processor -- start 2024-04-25T19:13:04.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.067+0000 7f8d4c5ee640 1 -- start start 2024-04-25T19:13:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4c5ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4406e410 0x7f8d441d5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4c5ee640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d4409cb60 0x7f8d441d7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4c5ee640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 0x7f8d441d77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4c5ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4406eba0 con 0x7f8d4406e410 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4c5ee640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4406ea20 con 0x7f8d4409d010 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4c5ee640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8d440cf120 con 0x7f8d4409cb60 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4a38a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4406e410 0x7f8d441d5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4a38a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4406e410 0x7f8d441d5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51136/0 (socket says 172.21.15.28:51136) 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.068+0000 7f8d4a38a640 1 -- 172.21.15.28:0/3842022496 learned_addr learned my addr 172.21.15.28:0/3842022496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4ab8b640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 0x7f8d441d77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d49b89640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d4409cb60 0x7f8d441d7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4a38a640 1 -- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d4409cb60 msgr2=0x7f8d441d7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4a38a640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d4409cb60 0x7f8d441d7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4a38a640 1 -- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 msgr2=0x7f8d441d77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4a38a640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 0x7f8d441d77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4a38a640 1 -- 172.21.15.28:0/3842022496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d441d61e0 con 0x7f8d4406e410 2024-04-25T19:13:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4ab8b640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 0x7f8d441d77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4a38a640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4406e410 0x7f8d441d5650 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8d4002a790 tx=0x7f8d400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d3affd640 1 -- 172.21.15.28:0/3842022496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40037040 con 0x7f8d4406e410 2024-04-25T19:13:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d3affd640 1 -- 172.21.15.28:0/3842022496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d40004230 con 0x7f8d4406e410 2024-04-25T19:13:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d441d64c0 con 0x7f8d4406e410 2024-04-25T19:13:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d3affd640 1 -- 172.21.15.28:0/3842022496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40003610 con 0x7f8d4406e410 2024-04-25T19:13:04.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.069+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d441d6a80 con 0x7f8d4406e410 2024-04-25T19:13:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.070+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d44050740 con 0x7f8d4406e410 2024-04-25T19:13:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.071+0000 7f8d3affd640 1 -- 172.21.15.28:0/3842022496 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8d400037b0 con 0x7f8d4406e410 2024-04-25T19:13:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.071+0000 7f8d3affd640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d280744b0 0x7f8d280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.071+0000 7f8d49b89640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d280744b0 0x7f8d280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.071+0000 7f8d3affd640 1 -- 172.21.15.28:0/3842022496 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6640+0+0 (secure 0 0 0) 0x7f8d400b64a0 con 0x7f8d4406e410 2024-04-25T19:13:04.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.072+0000 7f8d49b89640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d280744b0 0x7f8d280769c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8d440cecf0 tx=0x7f8d34009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.076+0000 7f8d3affd640 1 -- 172.21.15.28:0/3842022496 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d400809b0 con 0x7f8d4406e410 2024-04-25T19:13:04.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.262+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d44072130 con 0x7f8d4406e410 2024-04-25T19:13:04.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.264+0000 7f8d3affd640 1 -- 172.21.15.28:0/3842022496 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18904 (secure 0 0 0) 0x7f8d4003be10 con 0x7f8d4406e410 2024-04-25T19:13:04.266 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:04.266 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:03.631324+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d280744b0 msgr2=0x7f8d280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d280744b0 0x7f8d280769c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8d440cecf0 tx=0x7f8d34009040 comp rx=0 tx=0).stop 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4406e410 msgr2=0x7f8d441d5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4406e410 0x7f8d441d5650 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8d4002a790 tx=0x7f8d400046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 shutdown_connections 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d280744b0 0x7f8d280769c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d4409d010 0x7f8d441d77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d4409cb60 0x7f8d441d7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 --2- 172.21.15.28:0/3842022496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4406e410 0x7f8d441d5650 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 >> 172.21.15.28:0/3842022496 conn(0x7f8d440d0430 msgr2=0x7f8d440c94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 shutdown_connections 2024-04-25T19:13:04.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.266+0000 7f8d4c5ee640 1 -- 172.21.15.28:0/3842022496 wait complete. 2024-04-25T19:13:04.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:04.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.389+0000 7fc670042640 1 Processor -- start 2024-04-25T19:13:04.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.389+0000 7fc670042640 1 -- start start 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.390+0000 7fc670042640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc66815de90 0x7fc6680b7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.390+0000 7fc670042640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6680b1ae0 con 0x7fc6680b4130 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.390+0000 7fc670042640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6680b1d50 con 0x7fc668161140 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.390+0000 7fc670042640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc6680b23a0 con 0x7fc66815de90 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.390+0000 7fc66e5df640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc6680b4130 0x7fc66806c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40948/0 (socket says 172.21.15.28:40948) 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.390+0000 7fc66e5df640 1 -- 172.21.15.28:0/372610800 learned_addr learned my addr 172.21.15.28:0/372610800 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.390+0000 7fc66d5dd640 1 --2- 172.21.15.28:0/372610800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc66815de90 0x7fc6680b7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.391+0000 7fc66cddc640 1 -- 172.21.15.28:0/372610800 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536938852 0 0) 0x7fc6680b1ae0 con 0x7fc6680b4130 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.391+0000 7fc66cddc640 1 -- 172.21.15.28:0/372610800 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc654003520 con 0x7fc6680b4130 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.391+0000 7fc66cddc640 1 -- 172.21.15.28:0/372610800 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942069912 0 0) 0x7fc6680b1d50 con 0x7fc668161140 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.391+0000 7fc66cddc640 1 -- 172.21.15.28:0/372610800 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6680b1ae0 con 0x7fc668161140 2024-04-25T19:13:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.391+0000 7fc66d5dd640 1 -- 172.21.15.28:0/372610800 >> v1:172.21.15.53:6789/0 conn(0x7fc668161140 legacy=0x7fc6680b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.391+0000 7fc66d5dd640 1 -- 172.21.15.28:0/372610800 >> v1:172.21.15.28:6789/0 conn(0x7fc6680b4130 legacy=0x7fc66806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.391+0000 7fc66d5dd640 1 -- 172.21.15.28:0/372610800 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6680b2030 con 0x7fc66815de90 2024-04-25T19:13:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.392+0000 7fc66d5dd640 1 --2- 172.21.15.28:0/372610800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc66815de90 0x7fc6680b7800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc6580097d0 tx=0x7fc65802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=407bd1d9002dfc78 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.392+0000 7fc66cddc640 1 -- 172.21.15.28:0/372610800 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658037070 con 0x7fc66815de90 2024-04-25T19:13:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.392+0000 7fc66cddc640 1 -- 172.21.15.28:0/372610800 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc658032440 con 0x7fc66815de90 2024-04-25T19:13:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.392+0000 7fc66cddc640 1 -- 172.21.15.28:0/372610800 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6580327f0 con 0x7fc66815de90 2024-04-25T19:13:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.393+0000 7fc670042640 1 -- 172.21.15.28:0/372610800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc66815de90 msgr2=0x7fc6680b7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.393+0000 7fc670042640 1 --2- 172.21.15.28:0/372610800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc66815de90 0x7fc6680b7800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc6580097d0 tx=0x7fc65802a180 comp rx=0 tx=0).stop 2024-04-25T19:13:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.393+0000 7fc670042640 1 -- 172.21.15.28:0/372610800 shutdown_connections 2024-04-25T19:13:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.393+0000 7fc670042640 1 --2- 172.21.15.28:0/372610800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc66815de90 0x7fc6680b7800 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.393+0000 7fc670042640 1 -- 172.21.15.28:0/372610800 >> 172.21.15.28:0/372610800 conn(0x7fc668073d60 msgr2=0x7fc66809b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:04.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.393+0000 7fc670042640 1 -- 172.21.15.28:0/372610800 shutdown_connections 2024-04-25T19:13:04.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.393+0000 7fc670042640 1 -- 172.21.15.28:0/372610800 wait complete. 2024-04-25T19:13:04.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.394+0000 7fc670042640 1 Processor -- start 2024-04-25T19:13:04.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.394+0000 7fc670042640 1 -- start start 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc670042640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6680b4130 0x7fc6681557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc670042640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66815de90 0x7fc66815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc670042640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc66815f6e0 0x7fc66815faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc670042640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc6680b2830 con 0x7fc66815de90 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc670042640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc668075b80 con 0x7fc66815f6e0 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc670042640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc66815c100 con 0x7fc6680b4130 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66815de90 0x7fc66815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66815de90 0x7fc66815a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51148/0 (socket says 172.21.15.28:51148) 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 -- 172.21.15.28:0/1612831458 learned_addr learned my addr 172.21.15.28:0/1612831458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66ddde640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6680b4130 0x7fc6681557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66e5df640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc66815f6e0 0x7fc66815faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 -- 172.21.15.28:0/1612831458 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6680b4130 msgr2=0x7fc6681557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6680b4130 0x7fc6681557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 -- 172.21.15.28:0/1612831458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc66815f6e0 msgr2=0x7fc66815faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc66815f6e0 0x7fc66815faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 -- 172.21.15.28:0/1612831458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc668160030 con 0x7fc66815de90 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.396+0000 7fc66d5dd640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66815de90 0x7fc66815a5a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc658035040 tx=0x7fc6580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.397+0000 7fc65e7fc640 1 -- 172.21.15.28:0/1612831458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658041020 con 0x7fc66815de90 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.397+0000 7fc65e7fc640 1 -- 172.21.15.28:0/1612831458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc65803d6e0 con 0x7fc66815de90 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.397+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66815b060 con 0x7fc66815de90 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.397+0000 7fc65e7fc640 1 -- 172.21.15.28:0/1612831458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc658037040 con 0x7fc66815de90 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.397+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66815b620 con 0x7fc66815de90 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.398+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc66815add0 con 0x7fc66815de90 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.398+0000 7fc65e7fc640 1 -- 172.21.15.28:0/1612831458 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc658044020 con 0x7fc66815de90 2024-04-25T19:13:04.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.399+0000 7fc65e7fc640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc64c0744e0 0x7fc64c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.399+0000 7fc65e7fc640 1 -- 172.21.15.28:0/1612831458 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6640+0+0 (secure 0 0 0) 0x7fc6580b5980 con 0x7fc66815de90 2024-04-25T19:13:04.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.399+0000 7fc66ddde640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc64c0744e0 0x7fc64c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.399+0000 7fc66ddde640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc64c0744e0 0x7fc64c0769f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc664007fd0 tx=0x7fc664009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.404+0000 7fc65e7fc640 1 -- 172.21.15.28:0/1612831458 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc65804b070 con 0x7fc66815de90 2024-04-25T19:13:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.591+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc668155d00 con 0x7fc66815de90 2024-04-25T19:13:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.592+0000 7fc65e7fc640 1 -- 172.21.15.28:0/1612831458 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+18904 (secure 0 0 0) 0x7fc668155d00 con 0x7fc66815de90 2024-04-25T19:13:04.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:04.594 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":45,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:03.631324+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc64c0744e0 msgr2=0x7fc64c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc64c0744e0 0x7fc64c0769f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc664007fd0 tx=0x7fc664009040 comp rx=0 tx=0).stop 2024-04-25T19:13:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66815de90 msgr2=0x7fc66815a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66815de90 0x7fc66815a5a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc658035040 tx=0x7fc6580046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 shutdown_connections 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc64c0744e0 0x7fc64c0769f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc66815f6e0 0x7fc66815faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc66815de90 0x7fc66815a5a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 --2- 172.21.15.28:0/1612831458 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc6680b4130 0x7fc6681557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 >> 172.21.15.28:0/1612831458 conn(0x7fc668073d60 msgr2=0x7fc66806fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 shutdown_connections 2024-04-25T19:13:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.594+0000 7fc670042640 1 -- 172.21.15.28:0/1612831458 wait complete. 2024-04-25T19:13:04.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs standby_count_wanted 0 2024-04-25T19:13:04.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.720+0000 7f0f044d8640 1 Processor -- start 2024-04-25T19:13:04.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.720+0000 7f0f044d8640 1 -- start start 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f044d8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 0x7f0efc0d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f044d8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0efc0b79f0 0x7f0efc0b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f044d8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0efc160720 con 0x7f0efc08cce0 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f044d8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f0efc0b8340 con 0x7f0efc0b79f0 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f044d8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0efc0d5640 con 0x7f0efc160310 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f01a73640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0efc0b79f0 0x7f0efc0b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f01a73640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0efc0b79f0 0x7f0efc0b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51160/0 (socket says 172.21.15.28:51160) 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f01a73640 1 -- 172.21.15.28:0/3023292584 learned_addr learned my addr 172.21.15.28:0/3023292584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f02274640 1 --2- 172.21.15.28:0/3023292584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 0x7f0efc0d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f01a73640 1 -- 172.21.15.28:0/3023292584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 msgr2=0x7f0efc0d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f01a73640 1 --2- 172.21.15.28:0/3023292584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 0x7f0efc0d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f01a73640 1 -- 172.21.15.28:0/3023292584 >> v1:172.21.15.28:6789/0 conn(0x7f0efc08cce0 legacy=0x7f0efc08d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f01a73640 1 -- 172.21.15.28:0/3023292584 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc0d0f40 con 0x7f0efc0b79f0 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f01272640 1 -- 172.21.15.28:0/3023292584 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991799941 0 0) 0x7f0efc160720 con 0x7f0efc08cce0 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.722+0000 7f0f02274640 1 --2- 172.21.15.28:0/3023292584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 0x7f0efc0d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.723+0000 7f0f01a73640 1 --2- 172.21.15.28:0/3023292584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc0b79f0 0x7f0efc0b7e00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009870 tx=0x7f0ef002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf2083e2231e221 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.723+0000 7f0f01272640 1 -- 172.21.15.28:0/3023292584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0037070 con 0x7f0efc0b79f0 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.723+0000 7f0f01272640 1 -- 172.21.15.28:0/3023292584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef0032440 con 0x7f0efc0b79f0 2024-04-25T19:13:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.723+0000 7f0f01272640 1 -- 172.21.15.28:0/3023292584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0032810 con 0x7f0efc0b79f0 2024-04-25T19:13:04.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.724+0000 7f0f044d8640 1 -- 172.21.15.28:0/3023292584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc0b79f0 msgr2=0x7f0efc0b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.724+0000 7f0f044d8640 1 --2- 172.21.15.28:0/3023292584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc0b79f0 0x7f0efc0b7e00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009870 tx=0x7f0ef002a270 comp rx=0 tx=0).stop 2024-04-25T19:13:04.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.724+0000 7f0f044d8640 1 -- 172.21.15.28:0/3023292584 shutdown_connections 2024-04-25T19:13:04.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.724+0000 7f0f044d8640 1 --2- 172.21.15.28:0/3023292584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc0b79f0 0x7f0efc0b7e00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.724+0000 7f0f044d8640 1 --2- 172.21.15.28:0/3023292584 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 0x7f0efc0d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.724+0000 7f0f044d8640 1 -- 172.21.15.28:0/3023292584 >> 172.21.15.28:0/3023292584 conn(0x7f0efc0d2a50 msgr2=0x7f0efc0d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:04.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.724+0000 7f0f044d8640 1 -- 172.21.15.28:0/3023292584 shutdown_connections 2024-04-25T19:13:04.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.724+0000 7f0f044d8640 1 -- 172.21.15.28:0/3023292584 wait complete. 2024-04-25T19:13:04.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.726+0000 7f0f044d8640 1 Processor -- start 2024-04-25T19:13:04.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.726+0000 7f0f044d8640 1 -- start start 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f044d8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc08cce0 0x7f0efc0c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f044d8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0efc0b79f0 0x7f0efc14d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f044d8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 0x7f0efc14d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f044d8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc0b9400 con 0x7f0efc08cce0 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f044d8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0efc0d1500 con 0x7f0efc0b79f0 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f044d8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0efc0d0bf0 con 0x7f0efc160310 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f02274640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc08cce0 0x7f0efc0c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f02274640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc08cce0 0x7f0efc0c8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51176/0 (socket says 172.21.15.28:51176) 2024-04-25T19:13:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.727+0000 7f0f02274640 1 -- 172.21.15.28:0/1576118364 learned_addr learned my addr 172.21.15.28:0/1576118364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0f02274640 1 -- 172.21.15.28:0/1576118364 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 msgr2=0x7f0efc14d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0f02274640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 0x7f0efc14d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0f02274640 1 -- 172.21.15.28:0/1576118364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0efc0b79f0 msgr2=0x7f0efc14d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0f02274640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0efc0b79f0 0x7f0efc14d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0f02274640 1 -- 172.21.15.28:0/1576118364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0efc1e5660 con 0x7f0efc08cce0 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0f02274640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc08cce0 0x7f0efc0c8eb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400d7d0 tx=0x7f0ef400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0efaffd640 1 -- 172.21.15.28:0/1576118364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4004030 con 0x7f0efc08cce0 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0efaffd640 1 -- 172.21.15.28:0/1576118364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef40041d0 con 0x7f0efc08cce0 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0efc0ce6b0 con 0x7f0efc08cce0 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.728+0000 7f0efaffd640 1 -- 172.21.15.28:0/1576118364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef40096a0 con 0x7f0efc08cce0 2024-04-25T19:13:04.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.729+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0efc14b460 con 0x7f0efc08cce0 2024-04-25T19:13:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.729+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0efc068c30 con 0x7f0efc08cce0 2024-04-25T19:13:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.730+0000 7f0efaffd640 1 -- 172.21.15.28:0/1576118364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0ef4009840 con 0x7f0efc08cce0 2024-04-25T19:13:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.730+0000 7f0efaffd640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0ee80744e0 0x7f0ee80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.730+0000 7f0f01a73640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0ee80744e0 0x7f0ee80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.730+0000 7f0efaffd640 1 -- 172.21.15.28:0/1576118364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6640+0+0 (secure 0 0 0) 0x7f0ef40958b0 con 0x7f0efc08cce0 2024-04-25T19:13:04.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.731+0000 7f0f01a73640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0ee80744e0 0x7f0ee80769f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0034000 tx=0x7f0ef002c000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:04.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.735+0000 7f0efaffd640 1 -- 172.21.15.28:0/1576118364 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0efc068c30 con 0x7f0efc08cce0 2024-04-25T19:13:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:04.960+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0efc1e5830 con 0x7f0efc08cce0 2024-04-25T19:13:05.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.938+0000 7f0efaffd640 1 -- 172.21.15.28:0/1576118364 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "standby_count_wanted", "val": "0"}]=0 v24) ==== 123+0+0 (secure 0 0 0) 0x7f0ef4093e20 con 0x7f0efc08cce0 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0ee80744e0 msgr2=0x7f0ee80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0ee80744e0 0x7f0ee80769f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0034000 tx=0x7f0ef002c000 comp rx=0 tx=0).stop 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc08cce0 msgr2=0x7f0efc0c8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc08cce0 0x7f0efc0c8eb0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400d7d0 tx=0x7f0ef400dc90 comp rx=0 tx=0).stop 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 shutdown_connections 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0ee80744e0 0x7f0ee80769f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0efc160310 0x7f0efc14d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0efc0b79f0 0x7f0efc14d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 --2- 172.21.15.28:0/1576118364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0efc08cce0 0x7f0efc0c8eb0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:05.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 >> 172.21.15.28:0/1576118364 conn(0x7f0efc0d2a50 msgr2=0x7f0efc0bac80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:05.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 shutdown_connections 2024-04-25T19:13:05.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:05.941+0000 7f0f044d8640 1 -- 172.21.15.28:0/1576118364 wait complete. 2024-04-25T19:13:05.957 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T19:13:05.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs allow_standby_replay false 2024-04-25T19:13:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.077+0000 7f685006e640 1 Processor -- start 2024-04-25T19:13:06.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.077+0000 7f685006e640 1 -- start start 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f685006e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6848070b60 0x7f684815d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f685006e640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f684806fc30 0x7f6848070040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f685006e640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6848070580 con 0x7f6848160190 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f685006e640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f684815e0d0 con 0x7f684806fc30 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f685006e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6848153b00 con 0x7f6848070b60 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684e60b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6848160190 0x7f68481605a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40964/0 (socket says 172.21.15.28:40964) 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684e60b640 1 -- 172.21.15.28:0/3565671109 learned_addr learned my addr 172.21.15.28:0/3565671109 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684d609640 1 --2- 172.21.15.28:0/3565671109 >> v2:172.21.15.53:3300/0 conn(0x7f684806fc30 0x7f6848070040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684de0a640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6848070b60 0x7f684815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2497420687 0 0) 0x7f6840001110 con 0x7f6848160190 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6830003520 con 0x7f6848160190 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 176070384 0 0) 0x7f6830003520 con 0x7f6848160190 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6848070b60 msgr2=0x7f684815d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:06.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6848070b60 0x7f684815d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 >> v2:172.21.15.53:3300/0 conn(0x7f684806fc30 msgr2=0x7f6848070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 --2- 172.21.15.28:0/3565671109 >> v2:172.21.15.53:3300/0 conn(0x7f684806fc30 0x7f6848070040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68480af3c0 con 0x7f6848160190 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f6840001840 con 0x7f6848160190 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 >> v1:172.21.15.28:6789/0 conn(0x7f6848160190 legacy=0x7f68481605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6848070b60 0x7f68300039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6830003f30 0x7f6830004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830004d40 0x7f6830005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.079+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f684815e0d0 con 0x7f6830003f30 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6848153b00 con 0x7f6830004d40 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f683000bac0 con 0x7f6848070b60 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684d609640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6830003f30 0x7f6830004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684de0a640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6848070b60 0x7f68300039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684e60b640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830004d40 0x7f6830005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684d609640 1 -- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6848070b60 msgr2=0x7f68300039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:06.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684d609640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6848070b60 0x7f68300039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684d609640 1 -- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830004d40 msgr2=0x7f6830005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:06.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684d609640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830004d40 0x7f6830005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.080+0000 7f684d609640 1 -- 172.21.15.28:0/3565671109 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6830003520 con 0x7f6830003f30 2024-04-25T19:13:06.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.081+0000 7f684d609640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6830003f30 0x7f6830004800 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f68480cdfd0 tx=0x7f6830005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b55d4cc72b5def5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:06.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.081+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c036070 con 0x7f6830003f30 2024-04-25T19:13:06.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.081+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f683c031400 con 0x7f6830003f30 2024-04-25T19:13:06.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.081+0000 7f684ce08640 1 -- 172.21.15.28:0/3565671109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c009690 con 0x7f6830003f30 2024-04-25T19:13:06.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.081+0000 7f685006e640 1 -- 172.21.15.28:0/3565671109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6830003f30 msgr2=0x7f6830004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:06.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.081+0000 7f685006e640 1 --2- 172.21.15.28:0/3565671109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6830003f30 0x7f6830004800 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f68480cdfd0 tx=0x7f6830005830 comp rx=0 tx=0).stop 2024-04-25T19:13:06.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.082+0000 7f684e60b640 1 -- 172.21.15.28:0/3565671109 reap_dead start 2024-04-25T19:13:06.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.082+0000 7f685006e640 1 -- 172.21.15.28:0/3565671109 shutdown_connections 2024-04-25T19:13:06.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.082+0000 7f685006e640 1 -- 172.21.15.28:0/3565671109 >> 172.21.15.28:0/3565671109 conn(0x7f68480742c0 msgr2=0x7f684806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:06.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.082+0000 7f685006e640 1 -- 172.21.15.28:0/3565671109 shutdown_connections 2024-04-25T19:13:06.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.082+0000 7f685006e640 1 -- 172.21.15.28:0/3565671109 wait complete. 2024-04-25T19:13:06.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.083+0000 7f685006e640 1 Processor -- start 2024-04-25T19:13:06.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.084+0000 7f685006e640 1 -- start start 2024-04-25T19:13:06.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f685006e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830003f30 0x7f6848149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f685006e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68480d0cb0 0x7f68480cf2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f685006e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f684814b5d0 0x7f68480cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f685006e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f684815dd60 con 0x7f68480d0cb0 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f685006e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f68480af980 con 0x7f6830003f30 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f685006e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f684815f570 con 0x7f684814b5d0 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684d609640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68480d0cb0 0x7f68480cf2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684d609640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68480d0cb0 0x7f68480cf2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51198/0 (socket says 172.21.15.28:51198) 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684d609640 1 -- 172.21.15.28:0/145559217 learned_addr learned my addr 172.21.15.28:0/145559217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684e60b640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f684814b5d0 0x7f68480cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684de0a640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830003f30 0x7f6848149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684d609640 1 -- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f684814b5d0 msgr2=0x7f68480cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684d609640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f684814b5d0 0x7f68480cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684d609640 1 -- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830003f30 msgr2=0x7f6848149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684d609640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830003f30 0x7f6848149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684d609640 1 -- 172.21.15.28:0/145559217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684814a520 con 0x7f68480d0cb0 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.085+0000 7f684de0a640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830003f30 0x7f6848149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:06.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.086+0000 7f684d609640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68480d0cb0 0x7f68480cf2e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f683c009b50 tx=0x7f683c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:06.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.086+0000 7f68467fc640 1 -- 172.21.15.28:0/145559217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c040020 con 0x7f68480d0cb0 2024-04-25T19:13:06.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.086+0000 7f68467fc640 1 -- 172.21.15.28:0/145559217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f683c003a40 con 0x7f68480d0cb0 2024-04-25T19:13:06.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.086+0000 7f68467fc640 1 -- 172.21.15.28:0/145559217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683c036040 con 0x7f68480d0cb0 2024-04-25T19:13:06.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.086+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684814a810 con 0x7f68480d0cb0 2024-04-25T19:13:06.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.086+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f684814add0 con 0x7f68480d0cb0 2024-04-25T19:13:06.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.087+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68480cea00 con 0x7f68480d0cb0 2024-04-25T19:13:06.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.087+0000 7f68467fc640 1 -- 172.21.15.28:0/145559217 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f683c043020 con 0x7f68480d0cb0 2024-04-25T19:13:06.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.088+0000 7f68467fc640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f68340744e0 0x7f68340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:06.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.088+0000 7f68467fc640 1 -- 172.21.15.28:0/145559217 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x7f683c0b5860 con 0x7f68480d0cb0 2024-04-25T19:13:06.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.088+0000 7f684de0a640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f68340744e0 0x7f68340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:06.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.088+0000 7f684de0a640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f68340744e0 0x7f68340769f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6838009670 tx=0x7f683800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:06.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.092+0000 7f68467fc640 1 -- 172.21.15.28:0/145559217 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f683c04a070 con 0x7f68480d0cb0 2024-04-25T19:13:06.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.318+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f6848050740 con 0x7f68480d0cb0 2024-04-25T19:13:06.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.949+0000 7f68467fc640 1 -- 172.21.15.28:0/145559217 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "allow_standby_replay", "val": "false"}]=0 v25) ==== 127+0+0 (secure 0 0 0) 0x7f683c07fc60 con 0x7f68480d0cb0 2024-04-25T19:13:06.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f68340744e0 msgr2=0x7f68340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:06.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f68340744e0 0x7f68340769f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6838009670 tx=0x7f683800b040 comp rx=0 tx=0).stop 2024-04-25T19:13:06.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68480d0cb0 msgr2=0x7f68480cf2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:06.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68480d0cb0 0x7f68480cf2e0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f683c009b50 tx=0x7f683c004730 comp rx=0 tx=0).stop 2024-04-25T19:13:06.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 shutdown_connections 2024-04-25T19:13:06.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f68340744e0 0x7f68340769f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f684814b5d0 0x7f68480cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68480d0cb0 0x7f68480cf2e0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 --2- 172.21.15.28:0/145559217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6830003f30 0x7f6848149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:06.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 >> 172.21.15.28:0/145559217 conn(0x7f68480742c0 msgr2=0x7f684815d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:06.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 shutdown_connections 2024-04-25T19:13:06.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:06.952+0000 7f685006e640 1 -- 172.21.15.28:0/145559217 wait complete. 2024-04-25T19:13:06.967 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'recovery_fs' 2024-04-25T19:13:06.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.084+0000 7f03512ee640 1 Processor -- start 2024-04-25T19:13:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.084+0000 7f03512ee640 1 -- start start 2024-04-25T19:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f03512ee640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f034c0b5130 0x7f034c161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f03512ee640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f034c15f7d0 0x7f034c15fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f03512ee640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f034c163e80 con 0x7f034c06ded0 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f03512ee640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f034c163c30 con 0x7f034c15f7d0 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f03512ee640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f034c164a90 con 0x7f034c0b5130 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034b7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f034c15f7d0 0x7f034c15fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034a7fc640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f034c06ded0 0x7f034c06e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40972/0 (socket says 172.21.15.28:40972) 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034b7fe640 1 --2- 172.21.15.28:0/1578143399 >> v2:172.21.15.28:3300/0 conn(0x7f034c15f7d0 0x7f034c15fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51214/0 (socket says 172.21.15.28:51214) 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034a7fc640 1 -- 172.21.15.28:0/1578143399 learned_addr learned my addr 172.21.15.28:0/1578143399 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034affd640 1 --2- 172.21.15.28:0/1578143399 >> v2:172.21.15.53:3300/0 conn(0x7f034c0b5130 0x7f034c161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034b7fe640 1 -- 172.21.15.28:0/1578143399 >> v2:172.21.15.53:3300/0 conn(0x7f034c0b5130 msgr2=0x7f034c161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034b7fe640 1 --2- 172.21.15.28:0/1578143399 >> v2:172.21.15.53:3300/0 conn(0x7f034c0b5130 0x7f034c161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034b7fe640 1 -- 172.21.15.28:0/1578143399 >> v1:172.21.15.28:6789/0 conn(0x7f034c06ded0 legacy=0x7f034c06e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f0349ffb640 1 -- 172.21.15.28:0/1578143399 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244001301 0 0) 0x7f034c163e80 con 0x7f034c06ded0 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034b7fe640 1 -- 172.21.15.28:0/1578143399 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c164160 con 0x7f034c15f7d0 2024-04-25T19:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.086+0000 7f034affd640 1 --2- 172.21.15.28:0/1578143399 >> v2:172.21.15.53:3300/0 conn(0x7f034c0b5130 0x7f034c161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.087+0000 7f034b7fe640 1 --2- 172.21.15.28:0/1578143399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c15f7d0 0x7f034c15fbe0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f034c06ec40 tx=0x7f033c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0cee78bdc4f098e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.087+0000 7f0349ffb640 1 -- 172.21.15.28:0/1578143399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c007740 con 0x7f034c15f7d0 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.087+0000 7f0349ffb640 1 -- 172.21.15.28:0/1578143399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f033c0078e0 con 0x7f034c15f7d0 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.087+0000 7f0349ffb640 1 -- 172.21.15.28:0/1578143399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c007cb0 con 0x7f034c15f7d0 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.088+0000 7f03512ee640 1 -- 172.21.15.28:0/1578143399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c15f7d0 msgr2=0x7f034c15fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.088+0000 7f03512ee640 1 --2- 172.21.15.28:0/1578143399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c15f7d0 0x7f034c15fbe0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f034c06ec40 tx=0x7f033c02a680 comp rx=0 tx=0).stop 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.088+0000 7f03512ee640 1 -- 172.21.15.28:0/1578143399 shutdown_connections 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.088+0000 7f03512ee640 1 --2- 172.21.15.28:0/1578143399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c15f7d0 0x7f034c15fbe0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.088+0000 7f03512ee640 1 --2- 172.21.15.28:0/1578143399 >> v2:172.21.15.53:3300/0 conn(0x7f034c0b5130 0x7f034c161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.088+0000 7f03512ee640 1 -- 172.21.15.28:0/1578143399 >> 172.21.15.28:0/1578143399 conn(0x7f034c0b7ad0 msgr2=0x7f034c0b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.088+0000 7f03512ee640 1 -- 172.21.15.28:0/1578143399 shutdown_connections 2024-04-25T19:13:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.088+0000 7f03512ee640 1 -- 172.21.15.28:0/1578143399 wait complete. 2024-04-25T19:13:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.090+0000 7f03512ee640 1 Processor -- start 2024-04-25T19:13:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.090+0000 7f03512ee640 1 -- start start 2024-04-25T19:13:07.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f03512ee640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f034c06ded0 0x7f034c158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f03512ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c0b5130 0x7f034c15d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f03512ee640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f034c15f7d0 0x7f034c15da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f03512ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f034c160120 con 0x7f034c0b5130 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f03512ee640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f034c164720 con 0x7f034c15f7d0 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f03512ee640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f034c1602a0 con 0x7f034c06ded0 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f034a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c0b5130 0x7f034c15d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f034a7fc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c0b5130 0x7f034c15d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51222/0 (socket says 172.21.15.28:51222) 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.091+0000 7f034a7fc640 1 -- 172.21.15.28:0/391105355 learned_addr learned my addr 172.21.15.28:0/391105355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034affd640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f034c06ded0 0x7f034c158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034b7fe640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f034c15f7d0 0x7f034c15da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034a7fc640 1 -- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f034c06ded0 msgr2=0x7f034c158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034a7fc640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f034c06ded0 0x7f034c158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034a7fc640 1 -- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f034c15f7d0 msgr2=0x7f034c15da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034a7fc640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f034c15f7d0 0x7f034c15da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034a7fc640 1 -- 172.21.15.28:0/391105355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034c159f30 con 0x7f034c0b5130 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034b7fe640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f034c15f7d0 0x7f034c15da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034affd640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f034c06ded0 0x7f034c158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f034a7fc640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c0b5130 0x7f034c15d4d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f033400c460 tx=0x7f033400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f03437fe640 1 -- 172.21.15.28:0/391105355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334018070 con 0x7f034c0b5130 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f03437fe640 1 -- 172.21.15.28:0/391105355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0334007500 con 0x7f034c0b5130 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034c158ea0 con 0x7f034c0b5130 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f03437fe640 1 -- 172.21.15.28:0/391105355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334004a20 con 0x7f034c0b5130 2024-04-25T19:13:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.092+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034c159690 con 0x7f034c0b5130 2024-04-25T19:13:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.093+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f034c050740 con 0x7f034c0b5130 2024-04-25T19:13:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.094+0000 7f03437fe640 1 -- 172.21.15.28:0/391105355 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f033401b070 con 0x7f034c0b5130 2024-04-25T19:13:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.094+0000 7f03437fe640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f032c0744e0 0x7f032c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.094+0000 7f03437fe640 1 -- 172.21.15.28:0/391105355 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x7f0334094880 con 0x7f034c0b5130 2024-04-25T19:13:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.094+0000 7f034affd640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f032c0744e0 0x7f032c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.095+0000 7f034affd640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f032c0744e0 0x7f032c0769f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0344006fd0 tx=0x7f0344008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.099+0000 7f03437fe640 1 -- 172.21.15.28:0/391105355 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f033405ec70 con 0x7f034c0b5130 2024-04-25T19:13:07.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.320+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f034c1ec0f0 con 0x7f034c0b5130 2024-04-25T19:13:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.321+0000 7f03437fe640 1 -- 172.21.15.28:0/391105355 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+6468 (secure 0 0 0) 0x7f033405e570 con 0x7f034c0b5130 2024-04-25T19:13:07.323 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:07.323 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5589,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":25,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:06.319413+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T19:13:07.323 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-25T19:13:07.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f032c0744e0 msgr2=0x7f032c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f032c0744e0 0x7f032c0769f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0344006fd0 tx=0x7f0344008040 comp rx=0 tx=0).stop 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c0b5130 msgr2=0x7f034c15d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c0b5130 0x7f034c15d4d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f033400c460 tx=0x7f033400c920 comp rx=0 tx=0).stop 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 shutdown_connections 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f032c0744e0 0x7f032c0769f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f034c15f7d0 0x7f034c15da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f034c0b5130 0x7f034c15d4d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 --2- 172.21.15.28:0/391105355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f034c06ded0 0x7f034c158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 >> 172.21.15.28:0/391105355 conn(0x7f034c0b7ad0 msgr2=0x7f034c15edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 shutdown_connections 2024-04-25T19:13:07.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.323+0000 7f03512ee640 1 -- 172.21.15.28:0/391105355 wait complete. 2024-04-25T19:13:07.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.446+0000 7f2324642640 1 Processor -- start 2024-04-25T19:13:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.446+0000 7f2324642640 1 -- start start 2024-04-25T19:13:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.447+0000 7f2324642640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f231c0b5130 0x7f231c161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.447+0000 7f2324642640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f231c163e80 con 0x7f231c06ded0 2024-04-25T19:13:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.447+0000 7f2324642640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f231c164160 con 0x7f231c15f7d0 2024-04-25T19:13:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.447+0000 7f2324642640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f231c163c30 con 0x7f231c0b5130 2024-04-25T19:13:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.448+0000 7f23223de640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f231c0b5130 0x7f231c161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.448+0000 7f2321bdd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f231c06ded0 0x7f231c06e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40978/0 (socket says 172.21.15.28:40978) 2024-04-25T19:13:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.448+0000 7f2321bdd640 1 -- 172.21.15.28:0/2771016128 learned_addr learned my addr 172.21.15.28:0/2771016128 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.448+0000 7f23223de640 1 -- 172.21.15.28:0/2771016128 >> v1:172.21.15.53:6789/0 conn(0x7f231c15f7d0 legacy=0x7f231c15fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.448+0000 7f23223de640 1 -- 172.21.15.28:0/2771016128 >> v1:172.21.15.28:6789/0 conn(0x7f231c06ded0 legacy=0x7f231c06e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.448+0000 7f23223de640 1 -- 172.21.15.28:0/2771016128 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231c164440 con 0x7f231c0b5130 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.449+0000 7f23223de640 1 --2- 172.21.15.28:0/2771016128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c0b5130 0x7f231c161130 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f230c0097a0 tx=0x7f230c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5effedb5eca040a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.449+0000 7f23213dc640 1 -- 172.21.15.28:0/2771016128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c037070 con 0x7f231c0b5130 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.449+0000 7f23213dc640 1 -- 172.21.15.28:0/2771016128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f230c032440 con 0x7f231c0b5130 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.449+0000 7f23213dc640 1 -- 172.21.15.28:0/2771016128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f230c032810 con 0x7f231c0b5130 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.449+0000 7f2324642640 1 -- 172.21.15.28:0/2771016128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c0b5130 msgr2=0x7f231c161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.449+0000 7f2324642640 1 --2- 172.21.15.28:0/2771016128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c0b5130 0x7f231c161130 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f230c0097a0 tx=0x7f230c02a230 comp rx=0 tx=0).stop 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.450+0000 7f2324642640 1 -- 172.21.15.28:0/2771016128 shutdown_connections 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.450+0000 7f2324642640 1 --2- 172.21.15.28:0/2771016128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c0b5130 0x7f231c161130 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.450+0000 7f2324642640 1 -- 172.21.15.28:0/2771016128 >> 172.21.15.28:0/2771016128 conn(0x7f231c0b7ad0 msgr2=0x7f231c0b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.450+0000 7f2324642640 1 -- 172.21.15.28:0/2771016128 shutdown_connections 2024-04-25T19:13:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.450+0000 7f2324642640 1 -- 172.21.15.28:0/2771016128 wait complete. 2024-04-25T19:13:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.451+0000 7f2324642640 1 Processor -- start 2024-04-25T19:13:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.451+0000 7f2324642640 1 -- start start 2024-04-25T19:13:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2324642640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f231c06ded0 0x7f231c1f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2324642640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f231c0b5130 0x7f231c154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2324642640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c15f7d0 0x7f231c161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2324642640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f231c0d4910 con 0x7f231c15f7d0 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2324642640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f231c164a90 con 0x7f231c0b5130 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2324642640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f231c0d4a90 con 0x7f231c06ded0 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c15f7d0 0x7f231c161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c15f7d0 0x7f231c161ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51254/0 (socket says 172.21.15.28:51254) 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 -- 172.21.15.28:0/247794447 learned_addr learned my addr 172.21.15.28:0/247794447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f23223de640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f231c06ded0 0x7f231c1f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2321bdd640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f231c0b5130 0x7f231c154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 -- 172.21.15.28:0/247794447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f231c06ded0 msgr2=0x7f231c1f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f231c06ded0 0x7f231c1f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 -- 172.21.15.28:0/247794447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f231c0b5130 msgr2=0x7f231c154c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f231c0b5130 0x7f231c154c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 -- 172.21.15.28:0/247794447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231c157b00 con 0x7f231c15f7d0 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.453+0000 7f2322bdf640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c15f7d0 0x7f231c161ab0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f231400c6b0 tx=0x7f231400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.454+0000 7f2312ffd640 1 -- 172.21.15.28:0/247794447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314003b70 con 0x7f231c15f7d0 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.454+0000 7f2312ffd640 1 -- 172.21.15.28:0/247794447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2314003d30 con 0x7f231c15f7d0 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.454+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f231c157de0 con 0x7f231c15f7d0 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.454+0000 7f2312ffd640 1 -- 172.21.15.28:0/247794447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314005310 con 0x7f231c15f7d0 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.454+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f231c146d80 con 0x7f231c15f7d0 2024-04-25T19:13:07.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.455+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f231c050740 con 0x7f231c15f7d0 2024-04-25T19:13:07.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.455+0000 7f2312ffd640 1 -- 172.21.15.28:0/247794447 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f23140054b0 con 0x7f231c15f7d0 2024-04-25T19:13:07.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.456+0000 7f2312ffd640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23000744b0 0x7f23000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.456+0000 7f2312ffd640 1 -- 172.21.15.28:0/247794447 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x7f2314095200 con 0x7f231c15f7d0 2024-04-25T19:13:07.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.456+0000 7f23223de640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23000744b0 0x7f23000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.456+0000 7f23223de640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23000744b0 0x7f23000769c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f230c034000 tx=0x7f230c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.460+0000 7f2312ffd640 1 -- 172.21.15.28:0/247794447 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f231405efa0 con 0x7f231c15f7d0 2024-04-25T19:13:07.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.647+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f231c0d2fe0 con 0x7f231c15f7d0 2024-04-25T19:13:07.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.648+0000 7f2312ffd640 1 -- 172.21.15.28:0/247794447 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+18891 (secure 0 0 0) 0x7f23140632a0 con 0x7f231c15f7d0 2024-04-25T19:13:07.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:07.651 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":46,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:04.932308+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23000744b0 msgr2=0x7f23000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23000744b0 0x7f23000769c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f230c034000 tx=0x7f230c02e000 comp rx=0 tx=0).stop 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c15f7d0 msgr2=0x7f231c161ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c15f7d0 0x7f231c161ab0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f231400c6b0 tx=0x7f231400cb70 comp rx=0 tx=0).stop 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 shutdown_connections 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f23000744b0 0x7f23000769c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f231c15f7d0 0x7f231c161ab0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f231c0b5130 0x7f231c154c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 --2- 172.21.15.28:0/247794447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f231c06ded0 0x7f231c1f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 >> 172.21.15.28:0/247794447 conn(0x7f231c0b7ad0 msgr2=0x7f231c0d3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 shutdown_connections 2024-04-25T19:13:07.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.651+0000 7f2324642640 1 -- 172.21.15.28:0/247794447 wait complete. 2024-04-25T19:13:07.664 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T19:13:07.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:13:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.775+0000 7f8a7e114640 1 Processor -- start 2024-04-25T19:13:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.775+0000 7f8a7e114640 1 -- start start 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.776+0000 7f8a7e114640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a780b5130 0x7f8a78161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.776+0000 7f8a7e114640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 0x7f8a7815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.776+0000 7f8a7e114640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a78163e80 con 0x7f8a7806ded0 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.776+0000 7f8a7e114640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8a78163c30 con 0x7f8a780b5130 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.776+0000 7f8a7e114640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8a78164a90 con 0x7f8a7815f7d0 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.777+0000 7f8a777fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a780b5130 0x7f8a78161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.777+0000 7f8a777fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8a780b5130 0x7f8a78161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51264/0 (socket says 172.21.15.28:51264) 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.777+0000 7f8a777fe640 1 -- 172.21.15.28:0/62016603 learned_addr learned my addr 172.21.15.28:0/62016603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.777+0000 7f8a777fe640 1 -- 172.21.15.28:0/62016603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 msgr2=0x7f8a7815fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.777+0000 7f8a777fe640 1 --2- 172.21.15.28:0/62016603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 0x7f8a7815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.777+0000 7f8a777fe640 1 -- 172.21.15.28:0/62016603 >> v1:172.21.15.53:6789/0 conn(0x7f8a7806ded0 legacy=0x7f8a7806e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:07.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.777+0000 7f8a777fe640 1 -- 172.21.15.28:0/62016603 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a78164160 con 0x7f8a780b5130 2024-04-25T19:13:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.778+0000 7f8a777fe640 1 --2- 172.21.15.28:0/62016603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a780b5130 0x7f8a78161130 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8a7806ec40 tx=0x7f8a6802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610d132ebe9a02bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.778+0000 7f8a767fc640 1 -- 172.21.15.28:0/62016603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68036070 con 0x7f8a780b5130 2024-04-25T19:13:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.778+0000 7f8a767fc640 1 -- 172.21.15.28:0/62016603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68031600 con 0x7f8a780b5130 2024-04-25T19:13:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.778+0000 7f8a767fc640 1 -- 172.21.15.28:0/62016603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a680319d0 con 0x7f8a780b5130 2024-04-25T19:13:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.778+0000 7f8a7e114640 1 -- 172.21.15.28:0/62016603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a780b5130 msgr2=0x7f8a78161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.778+0000 7f8a7e114640 1 --2- 172.21.15.28:0/62016603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a780b5130 0x7f8a78161130 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8a7806ec40 tx=0x7f8a6802a6a0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.779+0000 7f8a7e114640 1 -- 172.21.15.28:0/62016603 shutdown_connections 2024-04-25T19:13:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.779+0000 7f8a7e114640 1 --2- 172.21.15.28:0/62016603 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 0x7f8a7815fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.779+0000 7f8a7e114640 1 --2- 172.21.15.28:0/62016603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a780b5130 0x7f8a78161130 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.779+0000 7f8a7e114640 1 -- 172.21.15.28:0/62016603 >> 172.21.15.28:0/62016603 conn(0x7f8a780b7ad0 msgr2=0x7f8a780b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.779+0000 7f8a7e114640 1 -- 172.21.15.28:0/62016603 shutdown_connections 2024-04-25T19:13:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.779+0000 7f8a7e114640 1 -- 172.21.15.28:0/62016603 wait complete. 2024-04-25T19:13:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.780+0000 7f8a7e114640 1 Processor -- start 2024-04-25T19:13:07.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.781+0000 7f8a7e114640 1 -- start start 2024-04-25T19:13:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a7e114640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a7806ded0 0x7f8a780ac510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a7e114640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780b5130 0x7f8a780c88f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a7e114640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 0x7f8a7814c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a7e114640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8a78160120 con 0x7f8a7806ded0 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a7e114640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8a78164720 con 0x7f8a780b5130 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a7e114640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8a781602a0 con 0x7f8a7815f7d0 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a7806ded0 0x7f8a780ac510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a777fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a7806ded0 0x7f8a780ac510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51274/0 (socket says 172.21.15.28:51274) 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a777fe640 1 -- 172.21.15.28:0/1150302083 learned_addr learned my addr 172.21.15.28:0/1150302083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a76ffd640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780b5130 0x7f8a780c88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a77fff640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 0x7f8a7814c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a777fe640 1 -- 172.21.15.28:0/1150302083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 msgr2=0x7f8a7814c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a777fe640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 0x7f8a7814c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a777fe640 1 -- 172.21.15.28:0/1150302083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780b5130 msgr2=0x7f8a780c88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a777fe640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780b5130 0x7f8a780c88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.782+0000 7f8a777fe640 1 -- 172.21.15.28:0/1150302083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a7814c760 con 0x7f8a7806ded0 2024-04-25T19:13:07.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.783+0000 7f8a777fe640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a7806ded0 0x7f8a780ac510 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8a68034040 tx=0x7f8a68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.783+0000 7f8a5ffff640 1 -- 172.21.15.28:0/1150302083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68041020 con 0x7f8a7806ded0 2024-04-25T19:13:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.783+0000 7f8a5ffff640 1 -- 172.21.15.28:0/1150302083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68007800 con 0x7f8a7806ded0 2024-04-25T19:13:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.783+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a780acb90 con 0x7f8a7806ded0 2024-04-25T19:13:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.783+0000 7f8a5ffff640 1 -- 172.21.15.28:0/1150302083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68036040 con 0x7f8a7806ded0 2024-04-25T19:13:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.783+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a780ad150 con 0x7f8a7806ded0 2024-04-25T19:13:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.784+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a78050740 con 0x7f8a7806ded0 2024-04-25T19:13:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.785+0000 7f8a5ffff640 1 -- 172.21.15.28:0/1150302083 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8a68044020 con 0x7f8a7806ded0 2024-04-25T19:13:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.785+0000 7f8a5ffff640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a640744b0 0x7f8a640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.785+0000 7f8a5ffff640 1 -- 172.21.15.28:0/1150302083 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x7f8a680b63b0 con 0x7f8a7806ded0 2024-04-25T19:13:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.785+0000 7f8a76ffd640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a640744b0 0x7f8a640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.786+0000 7f8a76ffd640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a640744b0 0x7f8a640769c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8a780d4160 tx=0x7f8a6c009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.790+0000 7f8a5ffff640 1 -- 172.21.15.28:0/1150302083 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8a6804b030 con 0x7f8a7806ded0 2024-04-25T19:13:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.972+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8a780d2fe0 con 0x7f8a640744b0 2024-04-25T19:13:07.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.981+0000 7f8a5ffff640 1 -- 172.21.15.28:0/1150302083 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332528 (secure 0 0 0) 0x7f8a780d2fe0 con 0x7f8a640744b0 2024-04-25T19:13:07.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:07.988 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:13:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a640744b0 msgr2=0x7f8a640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a640744b0 0x7f8a640769c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8a780d4160 tx=0x7f8a6c009060 comp rx=0 tx=0).stop 2024-04-25T19:13:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a7806ded0 msgr2=0x7f8a780ac510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:07.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a7806ded0 0x7f8a780ac510 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8a68034040 tx=0x7f8a68004a40 comp rx=0 tx=0).stop 2024-04-25T19:13:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 shutdown_connections 2024-04-25T19:13:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8a640744b0 0x7f8a640769c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8a7815f7d0 0x7f8a7814c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a780b5130 0x7f8a780c88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 --2- 172.21.15.28:0/1150302083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8a7806ded0 0x7f8a780ac510 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 >> 172.21.15.28:0/1150302083 conn(0x7f8a780b7ad0 msgr2=0x7f8a780b9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 shutdown_connections 2024-04-25T19:13:07.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:07.985+0000 7f8a7e114640 1 -- 172.21.15.28:0/1150302083 wait complete. 2024-04-25T19:13:07.998 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-25T19:13:06.857579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6881824,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":67,"num_read_kb":55,"num_write":203,"num_write_kb":7817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":660,"ondisk_log_size":660,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":250,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267064,"kb_used_data":17592,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754707656,"statfs":{"total":773094113280,"available":772820639744,"internally_reserved":0,"allocated":18014208,"data_stored":15688048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19545,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.945013"},"pg_stats":[{"pgid":"6.3f","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337205+0000","last_change":"2024-04-25T19:12:44.700653+0000","last_active":"2024-04-25T19:13:05.337205+0000","last_peered":"2024-04-25T19:13:05.337205+0000","last_clean":"2024-04-25T19:13:05.337205+0000","last_became_active":"2024-04-25T19:12:44.700326+0000","last_became_peered":"2024-04-25T19:12:44.700326+0000","last_unstale":"2024-04-25T19:13:05.337205+0000","last_undegraded":"2024-04-25T19:13:05.337205+0000","last_fullsized":"2024-04-25T19:13:05.337205+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:03:34.742465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"38'4","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336811+0000","last_change":"2024-04-25T19:12:44.699053+0000","last_active":"2024-04-25T19:13:05.336811+0000","last_peered":"2024-04-25T19:13:05.336811+0000","last_clean":"2024-04-25T19:13:05.336811+0000","last_became_active":"2024-04-25T19:12:44.698718+0000","last_became_peered":"2024-04-25T19:12:44.698718+0000","last_unstale":"2024-04-25T19:13:05.336811+0000","last_undegraded":"2024-04-25T19:13:05.336811+0000","last_fullsized":"2024-04-25T19:13:05.336811+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:05.953300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289673+0000","last_change":"2024-04-25T19:12:44.692595+0000","last_active":"2024-04-25T19:13:05.289673+0000","last_peered":"2024-04-25T19:13:05.289673+0000","last_clean":"2024-04-25T19:13:05.289673+0000","last_became_active":"2024-04-25T19:12:44.692288+0000","last_became_peered":"2024-04-25T19:12:44.692288+0000","last_unstale":"2024-04-25T19:13:05.289673+0000","last_undegraded":"2024-04-25T19:13:05.289673+0000","last_fullsized":"2024-04-25T19:13:05.289673+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:55:25.129409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"38'8","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336974+0000","last_change":"2024-04-25T19:12:44.691657+0000","last_active":"2024-04-25T19:13:05.336974+0000","last_peered":"2024-04-25T19:13:05.336974+0000","last_clean":"2024-04-25T19:13:05.336974+0000","last_became_active":"2024-04-25T19:12:44.691153+0000","last_became_peered":"2024-04-25T19:12:44.691153+0000","last_unstale":"2024-04-25T19:13:05.336974+0000","last_undegraded":"2024-04-25T19:13:05.336974+0000","last_fullsized":"2024-04-25T19:13:05.336974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:22.202361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"38'4","reported_seq":31,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337501+0000","last_change":"2024-04-25T19:12:44.685404+0000","last_active":"2024-04-25T19:13:05.337501+0000","last_peered":"2024-04-25T19:13:05.337501+0000","last_clean":"2024-04-25T19:13:05.337501+0000","last_became_active":"2024-04-25T19:12:44.685079+0000","last_became_peered":"2024-04-25T19:12:44.685079+0000","last_unstale":"2024-04-25T19:13:05.337501+0000","last_undegraded":"2024-04-25T19:13:05.337501+0000","last_fullsized":"2024-04-25T19:13:05.337501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:24.118674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"38'6","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338064+0000","last_change":"2024-04-25T19:12:44.702769+0000","last_active":"2024-04-25T19:13:05.338064+0000","last_peered":"2024-04-25T19:13:05.338064+0000","last_clean":"2024-04-25T19:13:05.338064+0000","last_became_active":"2024-04-25T19:12:44.702437+0000","last_became_peered":"2024-04-25T19:12:44.702437+0000","last_unstale":"2024-04-25T19:13:05.338064+0000","last_undegraded":"2024-04-25T19:13:05.338064+0000","last_fullsized":"2024-04-25T19:13:05.338064+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:22.449658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"38'5","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941003+0000","last_change":"2024-04-25T19:12:44.705815+0000","last_active":"2024-04-25T19:13:04.941003+0000","last_peered":"2024-04-25T19:13:04.941003+0000","last_clean":"2024-04-25T19:13:04.941003+0000","last_became_active":"2024-04-25T19:12:44.705457+0000","last_became_peered":"2024-04-25T19:12:44.705457+0000","last_unstale":"2024-04-25T19:13:04.941003+0000","last_undegraded":"2024-04-25T19:13:04.941003+0000","last_fullsized":"2024-04-25T19:13:04.941003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:27.321614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336344+0000","last_change":"2024-04-25T19:12:44.701020+0000","last_active":"2024-04-25T19:13:05.336344+0000","last_peered":"2024-04-25T19:13:05.336344+0000","last_clean":"2024-04-25T19:13:05.336344+0000","last_became_active":"2024-04-25T19:12:44.700672+0000","last_became_peered":"2024-04-25T19:12:44.700672+0000","last_unstale":"2024-04-25T19:13:05.336344+0000","last_undegraded":"2024-04-25T19:13:05.336344+0000","last_fullsized":"2024-04-25T19:13:05.336344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:31.406618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"38'11","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.290055+0000","last_change":"2024-04-25T19:12:44.693607+0000","last_active":"2024-04-25T19:13:05.290055+0000","last_peered":"2024-04-25T19:13:05.290055+0000","last_clean":"2024-04-25T19:13:05.290055+0000","last_became_active":"2024-04-25T19:12:44.693303+0000","last_became_peered":"2024-04-25T19:12:44.693303+0000","last_unstale":"2024-04-25T19:13:05.290055+0000","last_undegraded":"2024-04-25T19:13:05.290055+0000","last_fullsized":"2024-04-25T19:13:05.290055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:14.908856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"38'15","reported_seq":42,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940826+0000","last_change":"2024-04-25T19:12:44.695584+0000","last_active":"2024-04-25T19:13:04.940826+0000","last_peered":"2024-04-25T19:13:04.940826+0000","last_clean":"2024-04-25T19:13:04.940826+0000","last_became_active":"2024-04-25T19:12:44.695174+0000","last_became_peered":"2024-04-25T19:12:44.695174+0000","last_unstale":"2024-04-25T19:13:04.940826+0000","last_undegraded":"2024-04-25T19:13:04.940826+0000","last_fullsized":"2024-04-25T19:13:04.940826+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:18.897678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"38'6","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941939+0000","last_change":"2024-04-25T19:12:44.692511+0000","last_active":"2024-04-25T19:13:04.941939+0000","last_peered":"2024-04-25T19:13:04.941939+0000","last_clean":"2024-04-25T19:13:04.941939+0000","last_became_active":"2024-04-25T19:12:44.692175+0000","last_became_peered":"2024-04-25T19:12:44.692175+0000","last_unstale":"2024-04-25T19:13:04.941939+0000","last_undegraded":"2024-04-25T19:13:04.941939+0000","last_fullsized":"2024-04-25T19:13:04.941939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:49.760886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289765+0000","last_change":"2024-04-25T19:12:44.691012+0000","last_active":"2024-04-25T19:13:05.289765+0000","last_peered":"2024-04-25T19:13:05.289765+0000","last_clean":"2024-04-25T19:13:05.289765+0000","last_became_active":"2024-04-25T19:12:44.690627+0000","last_became_peered":"2024-04-25T19:12:44.690627+0000","last_unstale":"2024-04-25T19:13:05.289765+0000","last_undegraded":"2024-04-25T19:13:05.289765+0000","last_fullsized":"2024-04-25T19:13:05.289765+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:21.849594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"38'10","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941342+0000","last_change":"2024-04-25T19:12:44.696238+0000","last_active":"2024-04-25T19:13:04.941342+0000","last_peered":"2024-04-25T19:13:04.941342+0000","last_clean":"2024-04-25T19:13:04.941342+0000","last_became_active":"2024-04-25T19:12:44.695884+0000","last_became_peered":"2024-04-25T19:12:44.695884+0000","last_unstale":"2024-04-25T19:13:04.941342+0000","last_undegraded":"2024-04-25T19:13:04.941342+0000","last_fullsized":"2024-04-25T19:13:04.941342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:11.387331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942314+0000","last_change":"2024-04-25T19:12:44.690998+0000","last_active":"2024-04-25T19:13:04.942314+0000","last_peered":"2024-04-25T19:13:04.942314+0000","last_clean":"2024-04-25T19:13:04.942314+0000","last_became_active":"2024-04-25T19:12:44.690126+0000","last_became_peered":"2024-04-25T19:12:44.690126+0000","last_unstale":"2024-04-25T19:13:04.942314+0000","last_undegraded":"2024-04-25T19:13:04.942314+0000","last_fullsized":"2024-04-25T19:13:04.942314+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:29.311080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941243+0000","last_change":"2024-04-25T19:12:44.681433+0000","last_active":"2024-04-25T19:13:04.941243+0000","last_peered":"2024-04-25T19:13:04.941243+0000","last_clean":"2024-04-25T19:13:04.941243+0000","last_became_active":"2024-04-25T19:12:44.681100+0000","last_became_peered":"2024-04-25T19:12:44.681100+0000","last_unstale":"2024-04-25T19:13:04.941243+0000","last_undegraded":"2024-04-25T19:13:04.941243+0000","last_fullsized":"2024-04-25T19:13:04.941243+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:03:34.742465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941600+0000","last_change":"2024-04-25T19:12:44.701150+0000","last_active":"2024-04-25T19:13:04.941600+0000","last_peered":"2024-04-25T19:13:04.941600+0000","last_clean":"2024-04-25T19:13:04.941600+0000","last_became_active":"2024-04-25T19:12:44.699787+0000","last_became_peered":"2024-04-25T19:12:44.699787+0000","last_unstale":"2024-04-25T19:13:04.941600+0000","last_undegraded":"2024-04-25T19:13:04.941600+0000","last_fullsized":"2024-04-25T19:13:04.941600+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:24.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940597+0000","last_change":"2024-04-25T19:12:44.706630+0000","last_active":"2024-04-25T19:13:04.940597+0000","last_peered":"2024-04-25T19:13:04.940597+0000","last_clean":"2024-04-25T19:13:04.940597+0000","last_became_active":"2024-04-25T19:12:44.705055+0000","last_became_peered":"2024-04-25T19:12:44.705055+0000","last_unstale":"2024-04-25T19:13:04.940597+0000","last_undegraded":"2024-04-25T19:13:04.940597+0000","last_fullsized":"2024-04-25T19:13:04.940597+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:48:50.429366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"38'10","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336993+0000","last_change":"2024-04-25T19:12:44.683787+0000","last_active":"2024-04-25T19:13:05.336993+0000","last_peered":"2024-04-25T19:13:05.336993+0000","last_clean":"2024-04-25T19:13:05.336993+0000","last_became_active":"2024-04-25T19:12:44.683469+0000","last_became_peered":"2024-04-25T19:12:44.683469+0000","last_unstale":"2024-04-25T19:13:05.336993+0000","last_undegraded":"2024-04-25T19:13:05.336993+0000","last_fullsized":"2024-04-25T19:13:05.336993+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:54.272820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337488+0000","last_change":"2024-04-25T19:12:44.683518+0000","last_active":"2024-04-25T19:13:05.337488+0000","last_peered":"2024-04-25T19:13:05.337488+0000","last_clean":"2024-04-25T19:13:05.337488+0000","last_became_active":"2024-04-25T19:12:44.683194+0000","last_became_peered":"2024-04-25T19:12:44.683194+0000","last_unstale":"2024-04-25T19:13:05.337488+0000","last_undegraded":"2024-04-25T19:13:05.337488+0000","last_fullsized":"2024-04-25T19:13:05.337488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:37.425520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336950+0000","last_change":"2024-04-25T19:12:44.697957+0000","last_active":"2024-04-25T19:13:05.336950+0000","last_peered":"2024-04-25T19:13:05.336950+0000","last_clean":"2024-04-25T19:13:05.336950+0000","last_became_active":"2024-04-25T19:12:44.697633+0000","last_became_peered":"2024-04-25T19:12:44.697633+0000","last_unstale":"2024-04-25T19:13:05.336950+0000","last_undegraded":"2024-04-25T19:13:05.336950+0000","last_fullsized":"2024-04-25T19:13:05.336950+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:45.995608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"38'11","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941488+0000","last_change":"2024-04-25T19:12:44.690591+0000","last_active":"2024-04-25T19:13:04.941488+0000","last_peered":"2024-04-25T19:13:04.941488+0000","last_clean":"2024-04-25T19:13:04.941488+0000","last_became_active":"2024-04-25T19:12:44.690265+0000","last_became_peered":"2024-04-25T19:12:44.690265+0000","last_unstale":"2024-04-25T19:13:04.941488+0000","last_undegraded":"2024-04-25T19:13:04.941488+0000","last_fullsized":"2024-04-25T19:13:04.941488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:02:57.343073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942727+0000","last_change":"2024-04-25T19:12:44.683217+0000","last_active":"2024-04-25T19:13:04.942727+0000","last_peered":"2024-04-25T19:13:04.942727+0000","last_clean":"2024-04-25T19:13:04.942727+0000","last_became_active":"2024-04-25T19:12:44.682860+0000","last_became_peered":"2024-04-25T19:12:44.682860+0000","last_unstale":"2024-04-25T19:13:04.942727+0000","last_undegraded":"2024-04-25T19:13:04.942727+0000","last_fullsized":"2024-04-25T19:13:04.942727+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:12.662654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940882+0000","last_change":"2024-04-25T19:12:44.704757+0000","last_active":"2024-04-25T19:13:04.940882+0000","last_peered":"2024-04-25T19:13:04.940882+0000","last_clean":"2024-04-25T19:13:04.940882+0000","last_became_active":"2024-04-25T19:12:44.704345+0000","last_became_peered":"2024-04-25T19:12:44.704345+0000","last_unstale":"2024-04-25T19:13:04.940882+0000","last_undegraded":"2024-04-25T19:13:04.940882+0000","last_fullsized":"2024-04-25T19:13:04.940882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:34.452188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"38'6","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336642+0000","last_change":"2024-04-25T19:12:44.685369+0000","last_active":"2024-04-25T19:13:05.336642+0000","last_peered":"2024-04-25T19:13:05.336642+0000","last_clean":"2024-04-25T19:13:05.336642+0000","last_became_active":"2024-04-25T19:12:44.684749+0000","last_became_peered":"2024-04-25T19:12:44.684749+0000","last_unstale":"2024-04-25T19:13:05.336642+0000","last_undegraded":"2024-04-25T19:13:05.336642+0000","last_fullsized":"2024-04-25T19:13:05.336642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:34.463674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"38'2","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337207+0000","last_change":"2024-04-25T19:12:44.688826+0000","last_active":"2024-04-25T19:13:05.337207+0000","last_peered":"2024-04-25T19:13:05.337207+0000","last_clean":"2024-04-25T19:13:05.337207+0000","last_became_active":"2024-04-25T19:12:44.687859+0000","last_became_peered":"2024-04-25T19:12:44.687859+0000","last_unstale":"2024-04-25T19:13:05.337207+0000","last_undegraded":"2024-04-25T19:13:05.337207+0000","last_fullsized":"2024-04-25T19:13:05.337207+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:29.311080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"38'15","reported_seq":44,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338318+0000","last_change":"2024-04-25T19:12:44.698551+0000","last_active":"2024-04-25T19:13:05.338318+0000","last_peered":"2024-04-25T19:13:05.338318+0000","last_clean":"2024-04-25T19:13:05.338318+0000","last_became_active":"2024-04-25T19:12:44.698111+0000","last_became_peered":"2024-04-25T19:12:44.698111+0000","last_unstale":"2024-04-25T19:13:05.338318+0000","last_undegraded":"2024-04-25T19:13:05.338318+0000","last_fullsized":"2024-04-25T19:13:05.338318+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:20.222641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337072+0000","last_change":"2024-04-25T19:12:44.681953+0000","last_active":"2024-04-25T19:13:05.337072+0000","last_peered":"2024-04-25T19:13:05.337072+0000","last_clean":"2024-04-25T19:13:05.337072+0000","last_became_active":"2024-04-25T19:12:44.681238+0000","last_became_peered":"2024-04-25T19:12:44.681238+0000","last_unstale":"2024-04-25T19:13:05.337072+0000","last_undegraded":"2024-04-25T19:13:05.337072+0000","last_fullsized":"2024-04-25T19:13:05.337072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:05.288431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"38'4","reported_seq":31,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941489+0000","last_change":"2024-04-25T19:12:44.706575+0000","last_active":"2024-04-25T19:13:04.941489+0000","last_peered":"2024-04-25T19:13:04.941489+0000","last_clean":"2024-04-25T19:13:04.941489+0000","last_became_active":"2024-04-25T19:12:44.706268+0000","last_became_peered":"2024-04-25T19:12:44.706268+0000","last_unstale":"2024-04-25T19:13:04.941489+0000","last_undegraded":"2024-04-25T19:13:04.941489+0000","last_fullsized":"2024-04-25T19:13:04.941489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:01.862563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"38'7","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336594+0000","last_change":"2024-04-25T19:12:44.682967+0000","last_active":"2024-04-25T19:13:05.336594+0000","last_peered":"2024-04-25T19:13:05.336594+0000","last_clean":"2024-04-25T19:13:05.336594+0000","last_became_active":"2024-04-25T19:12:44.682477+0000","last_became_peered":"2024-04-25T19:12:44.682477+0000","last_unstale":"2024-04-25T19:13:05.336594+0000","last_undegraded":"2024-04-25T19:13:05.336594+0000","last_fullsized":"2024-04-25T19:13:05.336594+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:28.034410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336708+0000","last_change":"2024-04-25T19:12:44.699586+0000","last_active":"2024-04-25T19:13:05.336708+0000","last_peered":"2024-04-25T19:13:05.336708+0000","last_clean":"2024-04-25T19:13:05.336708+0000","last_became_active":"2024-04-25T19:12:44.699289+0000","last_became_peered":"2024-04-25T19:12:44.699289+0000","last_unstale":"2024-04-25T19:13:05.336708+0000","last_undegraded":"2024-04-25T19:13:05.336708+0000","last_fullsized":"2024-04-25T19:13:05.336708+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:57.582503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"38'12","reported_seq":40,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941024+0000","last_change":"2024-04-25T19:12:44.702160+0000","last_active":"2024-04-25T19:13:04.941024+0000","last_peered":"2024-04-25T19:13:04.941024+0000","last_clean":"2024-04-25T19:13:04.941024+0000","last_became_active":"2024-04-25T19:12:44.701815+0000","last_became_peered":"2024-04-25T19:12:44.701815+0000","last_unstale":"2024-04-25T19:13:04.941024+0000","last_undegraded":"2024-04-25T19:13:04.941024+0000","last_fullsized":"2024-04-25T19:13:04.941024+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:14.908856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"38'9","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337918+0000","last_change":"2024-04-25T19:12:44.691655+0000","last_active":"2024-04-25T19:13:05.337918+0000","last_peered":"2024-04-25T19:13:05.337918+0000","last_clean":"2024-04-25T19:13:05.337918+0000","last_became_active":"2024-04-25T19:12:44.691354+0000","last_became_peered":"2024-04-25T19:12:44.691354+0000","last_unstale":"2024-04-25T19:13:05.337918+0000","last_undegraded":"2024-04-25T19:13:05.337918+0000","last_fullsized":"2024-04-25T19:13:05.337918+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:12.662654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338522+0000","last_change":"2024-04-25T19:13:03.934638+0000","last_active":"2024-04-25T19:13:05.338522+0000","last_peered":"2024-04-25T19:13:05.338522+0000","last_clean":"2024-04-25T19:13:05.338522+0000","last_became_active":"2024-04-25T19:13:03.934312+0000","last_became_peered":"2024-04-25T19:13:03.934312+0000","last_unstale":"2024-04-25T19:13:05.338522+0000","last_undegraded":"2024-04-25T19:13:05.338522+0000","last_fullsized":"2024-04-25T19:13:05.338522+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:34.745079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940399+0000","last_change":"2024-04-25T19:12:44.705515+0000","last_active":"2024-04-25T19:13:04.940399+0000","last_peered":"2024-04-25T19:13:04.940399+0000","last_clean":"2024-04-25T19:13:04.940399+0000","last_became_active":"2024-04-25T19:12:44.703843+0000","last_became_peered":"2024-04-25T19:12:44.703843+0000","last_unstale":"2024-04-25T19:13:04.940399+0000","last_undegraded":"2024-04-25T19:13:04.940399+0000","last_fullsized":"2024-04-25T19:13:04.940399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:24:52.866281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"42'8","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337346+0000","last_change":"2024-04-25T19:12:43.687421+0000","last_active":"2024-04-25T19:13:05.337346+0000","last_peered":"2024-04-25T19:13:05.337346+0000","last_clean":"2024-04-25T19:13:05.337346+0000","last_became_active":"2024-04-25T19:12:43.687150+0000","last_became_peered":"2024-04-25T19:12:43.687150+0000","last_unstale":"2024-04-25T19:13:05.337346+0000","last_undegraded":"2024-04-25T19:13:05.337346+0000","last_fullsized":"2024-04-25T19:13:05.337346+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:22.014467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941267+0000","last_change":"2024-04-25T19:13:03.940120+0000","last_active":"2024-04-25T19:13:04.941267+0000","last_peered":"2024-04-25T19:13:04.941267+0000","last_clean":"2024-04-25T19:13:04.941267+0000","last_became_active":"2024-04-25T19:13:03.938104+0000","last_became_peered":"2024-04-25T19:13:03.938104+0000","last_unstale":"2024-04-25T19:13:04.941267+0000","last_undegraded":"2024-04-25T19:13:04.941267+0000","last_fullsized":"2024-04-25T19:13:04.941267+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:16:42.097709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"38'15","reported_seq":44,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338411+0000","last_change":"2024-04-25T19:12:44.698702+0000","last_active":"2024-04-25T19:13:05.338411+0000","last_peered":"2024-04-25T19:13:05.338411+0000","last_clean":"2024-04-25T19:13:05.338411+0000","last_became_active":"2024-04-25T19:12:44.698378+0000","last_became_peered":"2024-04-25T19:12:44.698378+0000","last_unstale":"2024-04-25T19:13:05.338411+0000","last_undegraded":"2024-04-25T19:13:05.338411+0000","last_fullsized":"2024-04-25T19:13:05.338411+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.290035+0000","last_change":"2024-04-25T19:12:43.687002+0000","last_active":"2024-04-25T19:13:05.290035+0000","last_peered":"2024-04-25T19:13:05.290035+0000","last_clean":"2024-04-25T19:13:05.290035+0000","last_became_active":"2024-04-25T19:12:43.686681+0000","last_became_peered":"2024-04-25T19:12:43.686681+0000","last_unstale":"2024-04-25T19:13:05.290035+0000","last_undegraded":"2024-04-25T19:13:05.290035+0000","last_fullsized":"2024-04-25T19:13:05.290035+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337164+0000","last_change":"2024-04-25T19:13:03.942982+0000","last_active":"2024-04-25T19:13:05.337164+0000","last_peered":"2024-04-25T19:13:05.337164+0000","last_clean":"2024-04-25T19:13:05.337164+0000","last_became_active":"2024-04-25T19:13:03.942661+0000","last_became_peered":"2024-04-25T19:13:03.942661+0000","last_unstale":"2024-04-25T19:13:05.337164+0000","last_undegraded":"2024-04-25T19:13:05.337164+0000","last_fullsized":"2024-04-25T19:13:05.337164+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:43.943430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"38'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942242+0000","last_change":"2024-04-25T19:12:44.688116+0000","last_active":"2024-04-25T19:13:04.942242+0000","last_peered":"2024-04-25T19:13:04.942242+0000","last_clean":"2024-04-25T19:13:04.942242+0000","last_became_active":"2024-04-25T19:12:44.687776+0000","last_became_peered":"2024-04-25T19:12:44.687776+0000","last_unstale":"2024-04-25T19:13:04.942242+0000","last_undegraded":"2024-04-25T19:13:04.942242+0000","last_fullsized":"2024-04-25T19:13:04.942242+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:45.995608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"42'3","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338591+0000","last_change":"2024-04-25T19:12:43.682264+0000","last_active":"2024-04-25T19:13:05.338591+0000","last_peered":"2024-04-25T19:13:05.338591+0000","last_clean":"2024-04-25T19:13:05.338591+0000","last_became_active":"2024-04-25T19:12:43.681918+0000","last_became_peered":"2024-04-25T19:12:43.681918+0000","last_unstale":"2024-04-25T19:13:05.338591+0000","last_undegraded":"2024-04-25T19:13:05.338591+0000","last_fullsized":"2024-04-25T19:13:05.338591+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:55.257918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338273+0000","last_change":"2024-04-25T19:13:03.938028+0000","last_active":"2024-04-25T19:13:05.338273+0000","last_peered":"2024-04-25T19:13:05.338273+0000","last_clean":"2024-04-25T19:13:05.338273+0000","last_became_active":"2024-04-25T19:13:03.937701+0000","last_became_peered":"2024-04-25T19:13:03.937701+0000","last_unstale":"2024-04-25T19:13:05.338273+0000","last_undegraded":"2024-04-25T19:13:05.338273+0000","last_fullsized":"2024-04-25T19:13:05.338273+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:53.707497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337065+0000","last_change":"2024-04-25T19:12:44.699758+0000","last_active":"2024-04-25T19:13:05.337065+0000","last_peered":"2024-04-25T19:13:05.337065+0000","last_clean":"2024-04-25T19:13:05.337065+0000","last_became_active":"2024-04-25T19:12:44.699294+0000","last_became_peered":"2024-04-25T19:12:44.699294+0000","last_unstale":"2024-04-25T19:13:05.337065+0000","last_undegraded":"2024-04-25T19:13:05.337065+0000","last_fullsized":"2024-04-25T19:13:05.337065+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:48:50.429366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"42'3","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941281+0000","last_change":"2024-04-25T19:12:43.690606+0000","last_active":"2024-04-25T19:13:04.941281+0000","last_peered":"2024-04-25T19:13:04.941281+0000","last_clean":"2024-04-25T19:13:04.941281+0000","last_became_active":"2024-04-25T19:12:43.690163+0000","last_became_peered":"2024-04-25T19:12:43.690163+0000","last_unstale":"2024-04-25T19:13:04.941281+0000","last_undegraded":"2024-04-25T19:13:04.941281+0000","last_fullsized":"2024-04-25T19:13:04.941281+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:44.282277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336514+0000","last_change":"2024-04-25T19:13:03.939229+0000","last_active":"2024-04-25T19:13:05.336514+0000","last_peered":"2024-04-25T19:13:05.336514+0000","last_clean":"2024-04-25T19:13:05.336514+0000","last_became_active":"2024-04-25T19:13:03.937472+0000","last_became_peered":"2024-04-25T19:13:03.937472+0000","last_unstale":"2024-04-25T19:13:05.336514+0000","last_undegraded":"2024-04-25T19:13:05.336514+0000","last_fullsized":"2024-04-25T19:13:05.336514+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:36.137436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"38'8","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289514+0000","last_change":"2024-04-25T19:12:44.692946+0000","last_active":"2024-04-25T19:13:05.289514+0000","last_peered":"2024-04-25T19:13:05.289514+0000","last_clean":"2024-04-25T19:13:05.289514+0000","last_became_active":"2024-04-25T19:12:44.692070+0000","last_became_peered":"2024-04-25T19:12:44.692070+0000","last_unstale":"2024-04-25T19:13:05.289514+0000","last_undegraded":"2024-04-25T19:13:05.289514+0000","last_fullsized":"2024-04-25T19:13:05.289514+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:36.123098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942097+0000","last_change":"2024-04-25T19:12:43.682355+0000","last_active":"2024-04-25T19:13:04.942097+0000","last_peered":"2024-04-25T19:13:04.942097+0000","last_clean":"2024-04-25T19:13:04.942097+0000","last_became_active":"2024-04-25T19:12:43.682041+0000","last_became_peered":"2024-04-25T19:12:43.682041+0000","last_unstale":"2024-04-25T19:13:04.942097+0000","last_undegraded":"2024-04-25T19:13:04.942097+0000","last_fullsized":"2024-04-25T19:13:04.942097+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:44.218685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289478+0000","last_change":"2024-04-25T19:13:03.942567+0000","last_active":"2024-04-25T19:13:05.289478+0000","last_peered":"2024-04-25T19:13:05.289478+0000","last_clean":"2024-04-25T19:13:05.289478+0000","last_became_active":"2024-04-25T19:13:03.942266+0000","last_became_peered":"2024-04-25T19:13:03.942266+0000","last_unstale":"2024-04-25T19:13:05.289478+0000","last_undegraded":"2024-04-25T19:13:05.289478+0000","last_fullsized":"2024-04-25T19:13:05.289478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:09.016195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"38'10","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941388+0000","last_change":"2024-04-25T19:12:44.691213+0000","last_active":"2024-04-25T19:13:04.941388+0000","last_peered":"2024-04-25T19:13:04.941388+0000","last_clean":"2024-04-25T19:13:04.941388+0000","last_became_active":"2024-04-25T19:12:44.690834+0000","last_became_peered":"2024-04-25T19:12:44.690834+0000","last_unstale":"2024-04-25T19:13:04.941388+0000","last_undegraded":"2024-04-25T19:13:04.941388+0000","last_fullsized":"2024-04-25T19:13:04.941388+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:29:16.564095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940110+0000","last_change":"2024-04-25T19:12:43.677641+0000","last_active":"2024-04-25T19:13:04.940110+0000","last_peered":"2024-04-25T19:13:04.940110+0000","last_clean":"2024-04-25T19:13:04.940110+0000","last_became_active":"2024-04-25T19:12:43.677313+0000","last_became_peered":"2024-04-25T19:12:43.677313+0000","last_unstale":"2024-04-25T19:13:04.940110+0000","last_undegraded":"2024-04-25T19:13:04.940110+0000","last_fullsized":"2024-04-25T19:13:04.940110+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:26.370845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289683+0000","last_change":"2024-04-25T19:13:03.937675+0000","last_active":"2024-04-25T19:13:05.289683+0000","last_peered":"2024-04-25T19:13:05.289683+0000","last_clean":"2024-04-25T19:13:05.289683+0000","last_became_active":"2024-04-25T19:13:03.937373+0000","last_became_peered":"2024-04-25T19:13:03.937373+0000","last_unstale":"2024-04-25T19:13:05.289683+0000","last_undegraded":"2024-04-25T19:13:05.289683+0000","last_fullsized":"2024-04-25T19:13:05.289683+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:30.949327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"38'7","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942663+0000","last_change":"2024-04-25T19:12:44.694711+0000","last_active":"2024-04-25T19:13:04.942663+0000","last_peered":"2024-04-25T19:13:04.942663+0000","last_clean":"2024-04-25T19:13:04.942663+0000","last_became_active":"2024-04-25T19:12:44.693493+0000","last_became_peered":"2024-04-25T19:12:44.693493+0000","last_unstale":"2024-04-25T19:13:04.942663+0000","last_undegraded":"2024-04-25T19:13:04.942663+0000","last_fullsized":"2024-04-25T19:13:04.942663+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338006+0000","last_change":"2024-04-25T19:12:43.691781+0000","last_active":"2024-04-25T19:13:05.338006+0000","last_peered":"2024-04-25T19:13:05.338006+0000","last_clean":"2024-04-25T19:13:05.338006+0000","last_became_active":"2024-04-25T19:12:43.691468+0000","last_became_peered":"2024-04-25T19:12:43.691468+0000","last_unstale":"2024-04-25T19:13:05.338006+0000","last_undegraded":"2024-04-25T19:13:05.338006+0000","last_fullsized":"2024-04-25T19:13:05.338006+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:26.370845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942242+0000","last_change":"2024-04-25T19:13:03.941101+0000","last_active":"2024-04-25T19:13:04.942242+0000","last_peered":"2024-04-25T19:13:04.942242+0000","last_clean":"2024-04-25T19:13:04.942242+0000","last_became_active":"2024-04-25T19:13:03.940729+0000","last_became_peered":"2024-04-25T19:13:03.940729+0000","last_unstale":"2024-04-25T19:13:04.942242+0000","last_undegraded":"2024-04-25T19:13:04.942242+0000","last_fullsized":"2024-04-25T19:13:04.942242+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:55:44.384718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336862+0000","last_change":"2024-04-25T19:12:44.703258+0000","last_active":"2024-04-25T19:13:05.336862+0000","last_peered":"2024-04-25T19:13:05.336862+0000","last_clean":"2024-04-25T19:13:05.336862+0000","last_became_active":"2024-04-25T19:12:44.702920+0000","last_became_peered":"2024-04-25T19:12:44.702920+0000","last_unstale":"2024-04-25T19:13:05.336862+0000","last_undegraded":"2024-04-25T19:13:05.336862+0000","last_fullsized":"2024-04-25T19:13:05.336862+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:29:16.564095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940206+0000","last_change":"2024-04-25T19:12:43.687531+0000","last_active":"2024-04-25T19:13:04.940206+0000","last_peered":"2024-04-25T19:13:04.940206+0000","last_clean":"2024-04-25T19:13:04.940206+0000","last_became_active":"2024-04-25T19:12:43.687230+0000","last_became_peered":"2024-04-25T19:12:43.687230+0000","last_unstale":"2024-04-25T19:13:04.940206+0000","last_undegraded":"2024-04-25T19:13:04.940206+0000","last_fullsized":"2024-04-25T19:13:04.940206+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:22.014467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940924+0000","last_change":"2024-04-25T19:13:03.935697+0000","last_active":"2024-04-25T19:13:04.940924+0000","last_peered":"2024-04-25T19:13:04.940924+0000","last_clean":"2024-04-25T19:13:04.940924+0000","last_became_active":"2024-04-25T19:13:03.935360+0000","last_became_peered":"2024-04-25T19:13:03.935360+0000","last_unstale":"2024-04-25T19:13:04.940924+0000","last_undegraded":"2024-04-25T19:13:04.940924+0000","last_fullsized":"2024-04-25T19:13:04.940924+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:29:35.819404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"38'11","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337109+0000","last_change":"2024-04-25T19:12:44.684353+0000","last_active":"2024-04-25T19:13:05.337109+0000","last_peered":"2024-04-25T19:13:05.337109+0000","last_clean":"2024-04-25T19:13:05.337109+0000","last_became_active":"2024-04-25T19:12:44.684024+0000","last_became_peered":"2024-04-25T19:12:44.684024+0000","last_unstale":"2024-04-25T19:13:05.337109+0000","last_undegraded":"2024-04-25T19:13:05.337109+0000","last_fullsized":"2024-04-25T19:13:05.337109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:45.225366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337340+0000","last_change":"2024-04-25T19:12:43.685597+0000","last_active":"2024-04-25T19:13:05.337340+0000","last_peered":"2024-04-25T19:13:05.337340+0000","last_clean":"2024-04-25T19:13:05.337340+0000","last_became_active":"2024-04-25T19:12:43.685245+0000","last_became_peered":"2024-04-25T19:12:43.685245+0000","last_unstale":"2024-04-25T19:13:05.337340+0000","last_undegraded":"2024-04-25T19:13:05.337340+0000","last_fullsized":"2024-04-25T19:13:05.337340+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:35.845516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337154+0000","last_change":"2024-04-25T19:13:03.935954+0000","last_active":"2024-04-25T19:13:05.337154+0000","last_peered":"2024-04-25T19:13:05.337154+0000","last_clean":"2024-04-25T19:13:05.337154+0000","last_became_active":"2024-04-25T19:13:03.935656+0000","last_became_peered":"2024-04-25T19:13:03.935656+0000","last_unstale":"2024-04-25T19:13:05.337154+0000","last_undegraded":"2024-04-25T19:13:05.337154+0000","last_fullsized":"2024-04-25T19:13:05.337154+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:13:16.514939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941239+0000","last_change":"2024-04-25T19:12:44.698686+0000","last_active":"2024-04-25T19:13:04.941239+0000","last_peered":"2024-04-25T19:13:04.941239+0000","last_clean":"2024-04-25T19:13:04.941239+0000","last_became_active":"2024-04-25T19:12:44.697877+0000","last_became_peered":"2024-04-25T19:12:44.697877+0000","last_unstale":"2024-04-25T19:13:04.941239+0000","last_undegraded":"2024-04-25T19:13:04.941239+0000","last_fullsized":"2024-04-25T19:13:04.941239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:08.672351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941558+0000","last_change":"2024-04-25T19:12:43.691155+0000","last_active":"2024-04-25T19:13:04.941558+0000","last_peered":"2024-04-25T19:13:04.941558+0000","last_clean":"2024-04-25T19:13:04.941558+0000","last_became_active":"2024-04-25T19:12:43.690735+0000","last_became_peered":"2024-04-25T19:12:43.690735+0000","last_unstale":"2024-04-25T19:13:04.941558+0000","last_undegraded":"2024-04-25T19:13:04.941558+0000","last_fullsized":"2024-04-25T19:13:04.941558+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:36.418839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940523+0000","last_change":"2024-04-25T19:13:03.941630+0000","last_active":"2024-04-25T19:13:04.940523+0000","last_peered":"2024-04-25T19:13:04.940523+0000","last_clean":"2024-04-25T19:13:04.940523+0000","last_became_active":"2024-04-25T19:13:03.941304+0000","last_became_peered":"2024-04-25T19:13:03.941304+0000","last_unstale":"2024-04-25T19:13:04.940523+0000","last_undegraded":"2024-04-25T19:13:04.940523+0000","last_fullsized":"2024-04-25T19:13:04.940523+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:09.016195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"38'10","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289882+0000","last_change":"2024-04-25T19:12:44.698058+0000","last_active":"2024-04-25T19:13:05.289882+0000","last_peered":"2024-04-25T19:13:05.289882+0000","last_clean":"2024-04-25T19:13:05.289882+0000","last_became_active":"2024-04-25T19:12:44.695384+0000","last_became_peered":"2024-04-25T19:12:44.695384+0000","last_unstale":"2024-04-25T19:13:05.289882+0000","last_undegraded":"2024-04-25T19:13:05.289882+0000","last_fullsized":"2024-04-25T19:13:05.289882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:24.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"42'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942246+0000","last_change":"2024-04-25T19:12:43.686664+0000","last_active":"2024-04-25T19:13:04.942246+0000","last_peered":"2024-04-25T19:13:04.942246+0000","last_clean":"2024-04-25T19:13:04.942246+0000","last_became_active":"2024-04-25T19:12:43.686375+0000","last_became_peered":"2024-04-25T19:12:43.686375+0000","last_unstale":"2024-04-25T19:13:04.942246+0000","last_undegraded":"2024-04-25T19:13:04.942246+0000","last_fullsized":"2024-04-25T19:13:04.942246+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:35.845516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941920+0000","last_change":"2024-04-25T19:13:03.942015+0000","last_active":"2024-04-25T19:13:04.941920+0000","last_peered":"2024-04-25T19:13:04.941920+0000","last_clean":"2024-04-25T19:13:04.941920+0000","last_became_active":"2024-04-25T19:13:03.941240+0000","last_became_peered":"2024-04-25T19:13:03.941240+0000","last_unstale":"2024-04-25T19:13:04.941920+0000","last_undegraded":"2024-04-25T19:13:04.941920+0000","last_fullsized":"2024-04-25T19:13:04.941920+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:41.104903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941947+0000","last_change":"2024-04-25T19:12:44.690212+0000","last_active":"2024-04-25T19:13:04.941947+0000","last_peered":"2024-04-25T19:13:04.941947+0000","last_clean":"2024-04-25T19:13:04.941947+0000","last_became_active":"2024-04-25T19:12:44.689709+0000","last_became_peered":"2024-04-25T19:12:44.689709+0000","last_unstale":"2024-04-25T19:13:04.941947+0000","last_undegraded":"2024-04-25T19:13:04.941947+0000","last_fullsized":"2024-04-25T19:13:04.941947+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"42'23","reported_seq":52,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337470+0000","last_change":"2024-04-25T19:12:43.689503+0000","last_active":"2024-04-25T19:13:05.337470+0000","last_peered":"2024-04-25T19:13:05.337470+0000","last_clean":"2024-04-25T19:13:05.337470+0000","last_became_active":"2024-04-25T19:12:43.689203+0000","last_became_peered":"2024-04-25T19:12:43.689203+0000","last_unstale":"2024-04-25T19:13:05.337470+0000","last_undegraded":"2024-04-25T19:13:05.337470+0000","last_fullsized":"2024-04-25T19:13:05.337470+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:08:45.991172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941389+0000","last_change":"2024-04-25T19:13:03.939799+0000","last_active":"2024-04-25T19:13:04.941389+0000","last_peered":"2024-04-25T19:13:04.941389+0000","last_clean":"2024-04-25T19:13:04.941389+0000","last_became_active":"2024-04-25T19:13:03.939429+0000","last_became_peered":"2024-04-25T19:13:03.939429+0000","last_unstale":"2024-04-25T19:13:04.941389+0000","last_undegraded":"2024-04-25T19:13:04.941389+0000","last_fullsized":"2024-04-25T19:13:04.941389+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:30.642640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"38'6","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289544+0000","last_change":"2024-04-25T19:12:44.693888+0000","last_active":"2024-04-25T19:13:05.289544+0000","last_peered":"2024-04-25T19:13:05.289544+0000","last_clean":"2024-04-25T19:13:05.289544+0000","last_became_active":"2024-04-25T19:12:44.691696+0000","last_became_peered":"2024-04-25T19:12:44.691696+0000","last_unstale":"2024-04-25T19:13:05.289544+0000","last_undegraded":"2024-04-25T19:13:05.289544+0000","last_fullsized":"2024-04-25T19:13:05.289544+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:24:52.866281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"42'5","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337179+0000","last_change":"2024-04-25T19:12:43.691713+0000","last_active":"2024-04-25T19:13:05.337179+0000","last_peered":"2024-04-25T19:13:05.337179+0000","last_clean":"2024-04-25T19:13:05.337179+0000","last_became_active":"2024-04-25T19:12:43.691375+0000","last_became_peered":"2024-04-25T19:12:43.691375+0000","last_unstale":"2024-04-25T19:13:05.337179+0000","last_undegraded":"2024-04-25T19:13:05.337179+0000","last_fullsized":"2024-04-25T19:13:05.337179+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:58.179259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336426+0000","last_change":"2024-04-25T19:13:03.939456+0000","last_active":"2024-04-25T19:13:05.336426+0000","last_peered":"2024-04-25T19:13:05.336426+0000","last_clean":"2024-04-25T19:13:05.336426+0000","last_became_active":"2024-04-25T19:13:03.939169+0000","last_became_peered":"2024-04-25T19:13:03.939169+0000","last_unstale":"2024-04-25T19:13:05.336426+0000","last_undegraded":"2024-04-25T19:13:05.336426+0000","last_fullsized":"2024-04-25T19:13:05.336426+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:16:42.097709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336810+0000","last_change":"2024-04-25T19:12:44.699213+0000","last_active":"2024-04-25T19:13:05.336810+0000","last_peered":"2024-04-25T19:13:05.336810+0000","last_clean":"2024-04-25T19:13:05.336810+0000","last_became_active":"2024-04-25T19:12:44.698893+0000","last_became_peered":"2024-04-25T19:12:44.698893+0000","last_unstale":"2024-04-25T19:13:05.336810+0000","last_undegraded":"2024-04-25T19:13:05.336810+0000","last_fullsized":"2024-04-25T19:13:05.336810+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:01.862563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338114+0000","last_change":"2024-04-25T19:12:43.691127+0000","last_active":"2024-04-25T19:13:05.338114+0000","last_peered":"2024-04-25T19:13:05.338114+0000","last_clean":"2024-04-25T19:13:05.338114+0000","last_became_active":"2024-04-25T19:12:43.690372+0000","last_became_peered":"2024-04-25T19:12:43.690372+0000","last_unstale":"2024-04-25T19:13:05.338114+0000","last_undegraded":"2024-04-25T19:13:05.338114+0000","last_fullsized":"2024-04-25T19:13:05.338114+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289962+0000","last_change":"2024-04-25T19:13:03.935754+0000","last_active":"2024-04-25T19:13:05.289962+0000","last_peered":"2024-04-25T19:13:05.289962+0000","last_clean":"2024-04-25T19:13:05.289962+0000","last_became_active":"2024-04-25T19:13:03.935442+0000","last_became_peered":"2024-04-25T19:13:03.935442+0000","last_unstale":"2024-04-25T19:13:05.289962+0000","last_undegraded":"2024-04-25T19:13:05.289962+0000","last_fullsized":"2024-04-25T19:13:05.289962+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:38.152987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"38'10","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940049+0000","last_change":"2024-04-25T19:12:44.698557+0000","last_active":"2024-04-25T19:13:04.940049+0000","last_peered":"2024-04-25T19:13:04.940049+0000","last_clean":"2024-04-25T19:13:04.940049+0000","last_became_active":"2024-04-25T19:12:44.698170+0000","last_became_peered":"2024-04-25T19:12:44.698170+0000","last_unstale":"2024-04-25T19:13:04.940049+0000","last_undegraded":"2024-04-25T19:13:04.940049+0000","last_fullsized":"2024-04-25T19:13:04.940049+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:46:17.053151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336583+0000","last_change":"2024-04-25T19:12:43.687570+0000","last_active":"2024-04-25T19:13:05.336583+0000","last_peered":"2024-04-25T19:13:05.336583+0000","last_clean":"2024-04-25T19:13:05.336583+0000","last_became_active":"2024-04-25T19:12:43.687216+0000","last_became_peered":"2024-04-25T19:12:43.687216+0000","last_unstale":"2024-04-25T19:13:05.336583+0000","last_undegraded":"2024-04-25T19:13:05.336583+0000","last_fullsized":"2024-04-25T19:13:05.336583+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:56.285928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336635+0000","last_change":"2024-04-25T19:13:03.938560+0000","last_active":"2024-04-25T19:13:05.336635+0000","last_peered":"2024-04-25T19:13:05.336635+0000","last_clean":"2024-04-25T19:13:05.336635+0000","last_became_active":"2024-04-25T19:13:03.936631+0000","last_became_peered":"2024-04-25T19:13:03.936631+0000","last_unstale":"2024-04-25T19:13:05.336635+0000","last_undegraded":"2024-04-25T19:13:05.336635+0000","last_fullsized":"2024-04-25T19:13:05.336635+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:14:00.227930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"38'15","reported_seq":43,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337001+0000","last_change":"2024-04-25T19:12:44.699754+0000","last_active":"2024-04-25T19:13:05.337001+0000","last_peered":"2024-04-25T19:13:05.337001+0000","last_clean":"2024-04-25T19:13:05.337001+0000","last_became_active":"2024-04-25T19:12:44.699422+0000","last_became_peered":"2024-04-25T19:12:44.699422+0000","last_unstale":"2024-04-25T19:13:05.337001+0000","last_undegraded":"2024-04-25T19:13:05.337001+0000","last_fullsized":"2024-04-25T19:13:05.337001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:11.694018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"42'15","reported_seq":44,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941843+0000","last_change":"2024-04-25T19:12:43.677420+0000","last_active":"2024-04-25T19:13:04.941843+0000","last_peered":"2024-04-25T19:13:04.941843+0000","last_clean":"2024-04-25T19:13:04.941843+0000","last_became_active":"2024-04-25T19:12:43.677108+0000","last_became_peered":"2024-04-25T19:12:43.677108+0000","last_unstale":"2024-04-25T19:13:04.941843+0000","last_undegraded":"2024-04-25T19:13:04.941843+0000","last_fullsized":"2024-04-25T19:13:04.941843+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:32:50.902627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336986+0000","last_change":"2024-04-25T19:13:03.932214+0000","last_active":"2024-04-25T19:13:05.336986+0000","last_peered":"2024-04-25T19:13:05.336986+0000","last_clean":"2024-04-25T19:13:05.336986+0000","last_became_active":"2024-04-25T19:13:03.931746+0000","last_became_peered":"2024-04-25T19:13:03.931746+0000","last_unstale":"2024-04-25T19:13:05.336986+0000","last_undegraded":"2024-04-25T19:13:05.336986+0000","last_fullsized":"2024-04-25T19:13:05.336986+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:55:44.384718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337345+0000","last_change":"2024-04-25T19:12:44.699717+0000","last_active":"2024-04-25T19:13:05.337345+0000","last_peered":"2024-04-25T19:13:05.337345+0000","last_clean":"2024-04-25T19:13:05.337345+0000","last_became_active":"2024-04-25T19:12:44.699361+0000","last_became_peered":"2024-04-25T19:12:44.699361+0000","last_unstale":"2024-04-25T19:13:05.337345+0000","last_undegraded":"2024-04-25T19:13:05.337345+0000","last_fullsized":"2024-04-25T19:13:05.337345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:27.321614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"42'2","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941746+0000","last_change":"2024-04-25T19:12:43.679044+0000","last_active":"2024-04-25T19:13:04.941746+0000","last_peered":"2024-04-25T19:13:04.941746+0000","last_clean":"2024-04-25T19:13:04.941746+0000","last_became_active":"2024-04-25T19:12:43.678723+0000","last_became_peered":"2024-04-25T19:12:43.678723+0000","last_unstale":"2024-04-25T19:13:04.941746+0000","last_undegraded":"2024-04-25T19:13:04.941746+0000","last_fullsized":"2024-04-25T19:13:04.941746+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:16.031539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336986+0000","last_change":"2024-04-25T19:13:03.939099+0000","last_active":"2024-04-25T19:13:05.336986+0000","last_peered":"2024-04-25T19:13:05.336986+0000","last_clean":"2024-04-25T19:13:05.336986+0000","last_became_active":"2024-04-25T19:13:03.938624+0000","last_became_peered":"2024-04-25T19:13:03.938624+0000","last_unstale":"2024-04-25T19:13:05.336986+0000","last_undegraded":"2024-04-25T19:13:05.336986+0000","last_fullsized":"2024-04-25T19:13:05.336986+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:41.104903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942111+0000","last_change":"2024-04-25T19:12:44.697124+0000","last_active":"2024-04-25T19:13:04.942111+0000","last_peered":"2024-04-25T19:13:04.942111+0000","last_clean":"2024-04-25T19:13:04.942111+0000","last_became_active":"2024-04-25T19:12:44.695351+0000","last_became_peered":"2024-04-25T19:12:44.695351+0000","last_unstale":"2024-04-25T19:13:04.942111+0000","last_undegraded":"2024-04-25T19:13:04.942111+0000","last_fullsized":"2024-04-25T19:13:04.942111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:11.694018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"42'3","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942407+0000","last_change":"2024-04-25T19:12:43.678051+0000","last_active":"2024-04-25T19:13:04.942407+0000","last_peered":"2024-04-25T19:13:04.942407+0000","last_clean":"2024-04-25T19:13:04.942407+0000","last_became_active":"2024-04-25T19:12:43.677640+0000","last_became_peered":"2024-04-25T19:12:43.677640+0000","last_unstale":"2024-04-25T19:13:04.942407+0000","last_undegraded":"2024-04-25T19:13:04.942407+0000","last_fullsized":"2024-04-25T19:13:04.942407+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:06.360629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289976+0000","last_change":"2024-04-25T19:13:03.939051+0000","last_active":"2024-04-25T19:13:05.289976+0000","last_peered":"2024-04-25T19:13:05.289976+0000","last_clean":"2024-04-25T19:13:05.289976+0000","last_became_active":"2024-04-25T19:13:03.938713+0000","last_became_peered":"2024-04-25T19:13:03.938713+0000","last_unstale":"2024-04-25T19:13:05.289976+0000","last_undegraded":"2024-04-25T19:13:05.289976+0000","last_fullsized":"2024-04-25T19:13:05.289976+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:16.598382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"38'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942178+0000","last_change":"2024-04-25T19:12:44.695748+0000","last_active":"2024-04-25T19:13:04.942178+0000","last_peered":"2024-04-25T19:13:04.942178+0000","last_clean":"2024-04-25T19:13:04.942178+0000","last_became_active":"2024-04-25T19:12:44.695447+0000","last_became_peered":"2024-04-25T19:12:44.695447+0000","last_unstale":"2024-04-25T19:13:04.942178+0000","last_undegraded":"2024-04-25T19:13:04.942178+0000","last_fullsized":"2024-04-25T19:13:04.942178+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:05.288431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"42'2","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337501+0000","last_change":"2024-04-25T19:12:43.684026+0000","last_active":"2024-04-25T19:13:05.337501+0000","last_peered":"2024-04-25T19:13:05.337501+0000","last_clean":"2024-04-25T19:13:05.337501+0000","last_became_active":"2024-04-25T19:12:43.683693+0000","last_became_peered":"2024-04-25T19:12:43.683693+0000","last_unstale":"2024-04-25T19:13:05.337501+0000","last_undegraded":"2024-04-25T19:13:05.337501+0000","last_fullsized":"2024-04-25T19:13:05.337501+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:14.924183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336996+0000","last_change":"2024-04-25T19:13:03.938890+0000","last_active":"2024-04-25T19:13:05.336996+0000","last_peered":"2024-04-25T19:13:05.336996+0000","last_clean":"2024-04-25T19:13:05.336996+0000","last_became_active":"2024-04-25T19:13:03.938593+0000","last_became_peered":"2024-04-25T19:13:03.938593+0000","last_unstale":"2024-04-25T19:13:05.336996+0000","last_undegraded":"2024-04-25T19:13:05.336996+0000","last_fullsized":"2024-04-25T19:13:05.336996+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:28.376365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"38'9","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337161+0000","last_change":"2024-04-25T19:12:44.701567+0000","last_active":"2024-04-25T19:13:05.337161+0000","last_peered":"2024-04-25T19:13:05.337161+0000","last_clean":"2024-04-25T19:13:05.337161+0000","last_became_active":"2024-04-25T19:12:44.701201+0000","last_became_peered":"2024-04-25T19:12:44.701201+0000","last_unstale":"2024-04-25T19:13:05.337161+0000","last_undegraded":"2024-04-25T19:13:05.337161+0000","last_fullsized":"2024-04-25T19:13:05.337161+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:00.405188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338081+0000","last_change":"2024-04-25T19:12:43.684026+0000","last_active":"2024-04-25T19:13:05.338081+0000","last_peered":"2024-04-25T19:13:05.338081+0000","last_clean":"2024-04-25T19:13:05.338081+0000","last_became_active":"2024-04-25T19:12:43.683700+0000","last_became_peered":"2024-04-25T19:12:43.683700+0000","last_unstale":"2024-04-25T19:13:05.338081+0000","last_undegraded":"2024-04-25T19:13:05.338081+0000","last_fullsized":"2024-04-25T19:13:05.338081+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:33.456993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289574+0000","last_change":"2024-04-25T19:13:03.936847+0000","last_active":"2024-04-25T19:13:05.289574+0000","last_peered":"2024-04-25T19:13:05.289574+0000","last_clean":"2024-04-25T19:13:05.289574+0000","last_became_active":"2024-04-25T19:13:03.936539+0000","last_became_peered":"2024-04-25T19:13:03.936539+0000","last_unstale":"2024-04-25T19:13:05.289574+0000","last_undegraded":"2024-04-25T19:13:05.289574+0000","last_fullsized":"2024-04-25T19:13:05.289574+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:29:35.819404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941992+0000","last_change":"2024-04-25T19:12:44.699373+0000","last_active":"2024-04-25T19:13:04.941992+0000","last_peered":"2024-04-25T19:13:04.941992+0000","last_clean":"2024-04-25T19:13:04.941992+0000","last_became_active":"2024-04-25T19:12:44.693763+0000","last_became_peered":"2024-04-25T19:12:44.693763+0000","last_unstale":"2024-04-25T19:13:04.941992+0000","last_undegraded":"2024-04-25T19:13:04.941992+0000","last_fullsized":"2024-04-25T19:13:04.941992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:15.489770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"42'4","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337554+0000","last_change":"2024-04-25T19:12:43.687714+0000","last_active":"2024-04-25T19:13:05.337554+0000","last_peered":"2024-04-25T19:13:05.337554+0000","last_clean":"2024-04-25T19:13:05.337554+0000","last_became_active":"2024-04-25T19:12:43.687375+0000","last_became_peered":"2024-04-25T19:12:43.687375+0000","last_unstale":"2024-04-25T19:13:05.337554+0000","last_undegraded":"2024-04-25T19:13:05.337554+0000","last_fullsized":"2024-04-25T19:13:05.337554+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:01.568678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336400+0000","last_change":"2024-04-25T19:13:03.933439+0000","last_active":"2024-04-25T19:13:05.336400+0000","last_peered":"2024-04-25T19:13:05.336400+0000","last_clean":"2024-04-25T19:13:05.336400+0000","last_became_active":"2024-04-25T19:13:03.933097+0000","last_became_peered":"2024-04-25T19:13:03.933097+0000","last_unstale":"2024-04-25T19:13:05.336400+0000","last_undegraded":"2024-04-25T19:13:05.336400+0000","last_fullsized":"2024-04-25T19:13:05.336400+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:24:18.441249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"38'2","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289601+0000","last_change":"2024-04-25T19:12:44.694953+0000","last_active":"2024-04-25T19:13:05.289601+0000","last_peered":"2024-04-25T19:13:05.289601+0000","last_clean":"2024-04-25T19:13:05.289601+0000","last_became_active":"2024-04-25T19:12:44.694607+0000","last_became_peered":"2024-04-25T19:12:44.694607+0000","last_unstale":"2024-04-25T19:13:05.289601+0000","last_undegraded":"2024-04-25T19:13:05.289601+0000","last_fullsized":"2024-04-25T19:13:05.289601+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:34.452188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337309+0000","last_change":"2024-04-25T19:12:43.688266+0000","last_active":"2024-04-25T19:13:05.337309+0000","last_peered":"2024-04-25T19:13:05.337309+0000","last_clean":"2024-04-25T19:13:05.337309+0000","last_became_active":"2024-04-25T19:12:43.687912+0000","last_became_peered":"2024-04-25T19:12:43.687912+0000","last_unstale":"2024-04-25T19:13:05.337309+0000","last_undegraded":"2024-04-25T19:13:05.337309+0000","last_fullsized":"2024-04-25T19:13:05.337309+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:54.337918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338296+0000","last_change":"2024-04-25T19:13:03.939466+0000","last_active":"2024-04-25T19:13:05.338296+0000","last_peered":"2024-04-25T19:13:05.338296+0000","last_clean":"2024-04-25T19:13:05.338296+0000","last_became_active":"2024-04-25T19:13:03.939167+0000","last_became_peered":"2024-04-25T19:13:03.939167+0000","last_unstale":"2024-04-25T19:13:05.338296+0000","last_undegraded":"2024-04-25T19:13:05.338296+0000","last_fullsized":"2024-04-25T19:13:05.338296+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:55.378407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"38'5","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940795+0000","last_change":"2024-04-25T19:12:44.687324+0000","last_active":"2024-04-25T19:13:04.940795+0000","last_peered":"2024-04-25T19:13:04.940795+0000","last_clean":"2024-04-25T19:13:04.940795+0000","last_became_active":"2024-04-25T19:12:44.686274+0000","last_became_peered":"2024-04-25T19:12:44.686274+0000","last_unstale":"2024-04-25T19:13:04.940795+0000","last_undegraded":"2024-04-25T19:13:04.940795+0000","last_fullsized":"2024-04-25T19:13:04.940795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:05.953300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941165+0000","last_change":"2024-04-25T19:12:43.687265+0000","last_active":"2024-04-25T19:13:04.941165+0000","last_peered":"2024-04-25T19:13:04.941165+0000","last_clean":"2024-04-25T19:13:04.941165+0000","last_became_active":"2024-04-25T19:12:43.686948+0000","last_became_peered":"2024-04-25T19:12:43.686948+0000","last_unstale":"2024-04-25T19:13:04.941165+0000","last_undegraded":"2024-04-25T19:13:04.941165+0000","last_fullsized":"2024-04-25T19:13:04.941165+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:55.257918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336484+0000","last_change":"2024-04-25T19:13:03.938813+0000","last_active":"2024-04-25T19:13:05.336484+0000","last_peered":"2024-04-25T19:13:05.336484+0000","last_clean":"2024-04-25T19:13:05.336484+0000","last_became_active":"2024-04-25T19:13:03.938503+0000","last_became_peered":"2024-04-25T19:13:03.938503+0000","last_unstale":"2024-04-25T19:13:05.336484+0000","last_undegraded":"2024-04-25T19:13:05.336484+0000","last_fullsized":"2024-04-25T19:13:05.336484+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:19.204323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337126+0000","last_change":"2024-04-25T19:12:44.706030+0000","last_active":"2024-04-25T19:13:05.337126+0000","last_peered":"2024-04-25T19:13:05.337126+0000","last_clean":"2024-04-25T19:13:05.337126+0000","last_became_active":"2024-04-25T19:12:44.704826+0000","last_became_peered":"2024-04-25T19:12:44.704826+0000","last_unstale":"2024-04-25T19:13:05.337126+0000","last_undegraded":"2024-04-25T19:13:05.337126+0000","last_fullsized":"2024-04-25T19:13:05.337126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:11.387331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336728+0000","last_change":"2024-04-25T19:12:43.688066+0000","last_active":"2024-04-25T19:13:05.336728+0000","last_peered":"2024-04-25T19:13:05.336728+0000","last_clean":"2024-04-25T19:13:05.336728+0000","last_became_active":"2024-04-25T19:12:43.687694+0000","last_became_peered":"2024-04-25T19:12:43.687694+0000","last_unstale":"2024-04-25T19:13:05.336728+0000","last_undegraded":"2024-04-25T19:13:05.336728+0000","last_fullsized":"2024-04-25T19:13:05.336728+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:24.803044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941093+0000","last_change":"2024-04-25T19:13:03.936444+0000","last_active":"2024-04-25T19:13:04.941093+0000","last_peered":"2024-04-25T19:13:04.941093+0000","last_clean":"2024-04-25T19:13:04.941093+0000","last_became_active":"2024-04-25T19:13:03.936109+0000","last_became_peered":"2024-04-25T19:13:03.936109+0000","last_unstale":"2024-04-25T19:13:04.941093+0000","last_undegraded":"2024-04-25T19:13:04.941093+0000","last_fullsized":"2024-04-25T19:13:04.941093+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:30.949327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"38'9","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338016+0000","last_change":"2024-04-25T19:12:44.698861+0000","last_active":"2024-04-25T19:13:05.338016+0000","last_peered":"2024-04-25T19:13:05.338016+0000","last_clean":"2024-04-25T19:13:05.338016+0000","last_became_active":"2024-04-25T19:12:44.698178+0000","last_became_peered":"2024-04-25T19:12:44.698178+0000","last_unstale":"2024-04-25T19:13:05.338016+0000","last_undegraded":"2024-04-25T19:13:05.338016+0000","last_fullsized":"2024-04-25T19:13:05.338016+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:17.267296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"42'3","reported_seq":31,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941845+0000","last_change":"2024-04-25T19:12:43.685799+0000","last_active":"2024-04-25T19:13:04.941845+0000","last_peered":"2024-04-25T19:13:04.941845+0000","last_clean":"2024-04-25T19:13:04.941845+0000","last_became_active":"2024-04-25T19:12:43.685478+0000","last_became_peered":"2024-04-25T19:12:43.685478+0000","last_unstale":"2024-04-25T19:13:04.941845+0000","last_undegraded":"2024-04-25T19:13:04.941845+0000","last_fullsized":"2024-04-25T19:13:04.941845+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:21.195681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289842+0000","last_change":"2024-04-25T19:13:03.941282+0000","last_active":"2024-04-25T19:13:05.289842+0000","last_peered":"2024-04-25T19:13:05.289842+0000","last_clean":"2024-04-25T19:13:05.289842+0000","last_became_active":"2024-04-25T19:13:03.940988+0000","last_became_peered":"2024-04-25T19:13:03.940988+0000","last_unstale":"2024-04-25T19:13:05.289842+0000","last_undegraded":"2024-04-25T19:13:05.289842+0000","last_fullsized":"2024-04-25T19:13:05.289842+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:30.642640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"38'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941361+0000","last_change":"2024-04-25T19:12:44.699033+0000","last_active":"2024-04-25T19:13:04.941361+0000","last_peered":"2024-04-25T19:13:04.941361+0000","last_clean":"2024-04-25T19:13:04.941361+0000","last_became_active":"2024-04-25T19:12:44.698376+0000","last_became_peered":"2024-04-25T19:12:44.698376+0000","last_unstale":"2024-04-25T19:13:04.941361+0000","last_undegraded":"2024-04-25T19:13:04.941361+0000","last_fullsized":"2024-04-25T19:13:04.941361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:31.406618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"42'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336945+0000","last_change":"2024-04-25T19:12:43.688186+0000","last_active":"2024-04-25T19:13:05.336945+0000","last_peered":"2024-04-25T19:13:05.336945+0000","last_clean":"2024-04-25T19:13:05.336945+0000","last_became_active":"2024-04-25T19:12:43.687361+0000","last_became_peered":"2024-04-25T19:12:43.687361+0000","last_unstale":"2024-04-25T19:13:05.336945+0000","last_undegraded":"2024-04-25T19:13:05.336945+0000","last_fullsized":"2024-04-25T19:13:05.336945+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:37.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337294+0000","last_change":"2024-04-25T19:13:03.942670+0000","last_active":"2024-04-25T19:13:05.337294+0000","last_peered":"2024-04-25T19:13:05.337294+0000","last_clean":"2024-04-25T19:13:05.337294+0000","last_became_active":"2024-04-25T19:13:03.942221+0000","last_became_peered":"2024-04-25T19:13:03.942221+0000","last_unstale":"2024-04-25T19:13:05.337294+0000","last_undegraded":"2024-04-25T19:13:05.337294+0000","last_fullsized":"2024-04-25T19:13:05.337294+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:53.707497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"38'7","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336912+0000","last_change":"2024-04-25T19:12:44.705586+0000","last_active":"2024-04-25T19:13:05.336912+0000","last_peered":"2024-04-25T19:13:05.336912+0000","last_clean":"2024-04-25T19:13:05.336912+0000","last_became_active":"2024-04-25T19:12:44.704667+0000","last_became_peered":"2024-04-25T19:12:44.704667+0000","last_unstale":"2024-04-25T19:13:05.336912+0000","last_undegraded":"2024-04-25T19:13:05.336912+0000","last_fullsized":"2024-04-25T19:13:05.336912+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:49.760886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289740+0000","last_change":"2024-04-25T19:12:43.685345+0000","last_active":"2024-04-25T19:13:05.289740+0000","last_peered":"2024-04-25T19:13:05.289740+0000","last_clean":"2024-04-25T19:13:05.289740+0000","last_became_active":"2024-04-25T19:12:43.684968+0000","last_became_peered":"2024-04-25T19:12:43.684968+0000","last_unstale":"2024-04-25T19:13:05.289740+0000","last_undegraded":"2024-04-25T19:13:05.289740+0000","last_fullsized":"2024-04-25T19:13:05.289740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:33.456993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940523+0000","last_change":"2024-04-25T19:13:03.941152+0000","last_active":"2024-04-25T19:13:04.940523+0000","last_peered":"2024-04-25T19:13:04.940523+0000","last_clean":"2024-04-25T19:13:04.940523+0000","last_became_active":"2024-04-25T19:13:03.940693+0000","last_became_peered":"2024-04-25T19:13:03.940693+0000","last_unstale":"2024-04-25T19:13:04.940523+0000","last_undegraded":"2024-04-25T19:13:04.940523+0000","last_fullsized":"2024-04-25T19:13:04.940523+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:14:00.227930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"38'8","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289674+0000","last_change":"2024-04-25T19:12:44.702919+0000","last_active":"2024-04-25T19:13:05.289674+0000","last_peered":"2024-04-25T19:13:05.289674+0000","last_clean":"2024-04-25T19:13:05.289674+0000","last_became_active":"2024-04-25T19:12:44.702600+0000","last_became_peered":"2024-04-25T19:12:44.702600+0000","last_unstale":"2024-04-25T19:13:05.289674+0000","last_undegraded":"2024-04-25T19:13:05.289674+0000","last_fullsized":"2024-04-25T19:13:05.289674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:15.489770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.290049+0000","last_change":"2024-04-25T19:12:43.689072+0000","last_active":"2024-04-25T19:13:05.290049+0000","last_peered":"2024-04-25T19:13:05.290049+0000","last_clean":"2024-04-25T19:13:05.290049+0000","last_became_active":"2024-04-25T19:12:43.688665+0000","last_became_peered":"2024-04-25T19:12:43.688665+0000","last_unstale":"2024-04-25T19:13:05.290049+0000","last_undegraded":"2024-04-25T19:13:05.290049+0000","last_fullsized":"2024-04-25T19:13:05.290049+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:23.111993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940943+0000","last_change":"2024-04-25T19:13:03.937565+0000","last_active":"2024-04-25T19:13:04.940943+0000","last_peered":"2024-04-25T19:13:04.940943+0000","last_clean":"2024-04-25T19:13:04.940943+0000","last_became_active":"2024-04-25T19:13:03.937259+0000","last_became_peered":"2024-04-25T19:13:03.937259+0000","last_unstale":"2024-04-25T19:13:04.940943+0000","last_undegraded":"2024-04-25T19:13:04.940943+0000","last_fullsized":"2024-04-25T19:13:04.940943+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:41.764632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"38'5","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337977+0000","last_change":"2024-04-25T19:12:44.683442+0000","last_active":"2024-04-25T19:13:05.337977+0000","last_peered":"2024-04-25T19:13:05.337977+0000","last_clean":"2024-04-25T19:13:05.337977+0000","last_became_active":"2024-04-25T19:12:44.683137+0000","last_became_peered":"2024-04-25T19:12:44.683137+0000","last_unstale":"2024-04-25T19:13:05.337977+0000","last_undegraded":"2024-04-25T19:13:05.337977+0000","last_fullsized":"2024-04-25T19:13:05.337977+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:16.882127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336777+0000","last_change":"2024-04-25T19:12:43.688566+0000","last_active":"2024-04-25T19:13:05.336777+0000","last_peered":"2024-04-25T19:13:05.336777+0000","last_clean":"2024-04-25T19:13:05.336777+0000","last_became_active":"2024-04-25T19:12:43.688009+0000","last_became_peered":"2024-04-25T19:12:43.688009+0000","last_unstale":"2024-04-25T19:13:05.336777+0000","last_undegraded":"2024-04-25T19:13:05.336777+0000","last_fullsized":"2024-04-25T19:13:05.336777+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:47.259986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337139+0000","last_change":"2024-04-25T19:13:03.938051+0000","last_active":"2024-04-25T19:13:05.337139+0000","last_peered":"2024-04-25T19:13:05.337139+0000","last_clean":"2024-04-25T19:13:05.337139+0000","last_became_active":"2024-04-25T19:13:03.937743+0000","last_became_peered":"2024-04-25T19:13:03.937743+0000","last_unstale":"2024-04-25T19:13:05.337139+0000","last_undegraded":"2024-04-25T19:13:05.337139+0000","last_fullsized":"2024-04-25T19:13:05.337139+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:15.317830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942485+0000","last_change":"2024-04-25T19:12:44.704959+0000","last_active":"2024-04-25T19:13:04.942485+0000","last_peered":"2024-04-25T19:13:04.942485+0000","last_clean":"2024-04-25T19:13:04.942485+0000","last_became_active":"2024-04-25T19:12:44.704200+0000","last_became_peered":"2024-04-25T19:12:44.704200+0000","last_unstale":"2024-04-25T19:13:04.942485+0000","last_undegraded":"2024-04-25T19:13:04.942485+0000","last_fullsized":"2024-04-25T19:13:04.942485+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:57.582503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940596+0000","last_change":"2024-04-25T19:12:43.686166+0000","last_active":"2024-04-25T19:13:04.940596+0000","last_peered":"2024-04-25T19:13:04.940596+0000","last_clean":"2024-04-25T19:13:04.940596+0000","last_became_active":"2024-04-25T19:12:43.685724+0000","last_became_peered":"2024-04-25T19:12:43.685724+0000","last_unstale":"2024-04-25T19:13:04.940596+0000","last_undegraded":"2024-04-25T19:13:04.940596+0000","last_fullsized":"2024-04-25T19:13:04.940596+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:00:37.087793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336585+0000","last_change":"2024-04-25T19:13:03.933206+0000","last_active":"2024-04-25T19:13:05.336585+0000","last_peered":"2024-04-25T19:13:05.336585+0000","last_clean":"2024-04-25T19:13:05.336585+0000","last_became_active":"2024-04-25T19:13:03.932874+0000","last_became_peered":"2024-04-25T19:13:03.932874+0000","last_unstale":"2024-04-25T19:13:05.336585+0000","last_undegraded":"2024-04-25T19:13:05.336585+0000","last_fullsized":"2024-04-25T19:13:05.336585+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:18.325268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"38'7","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940638+0000","last_change":"2024-04-25T19:12:44.702272+0000","last_active":"2024-04-25T19:13:04.940638+0000","last_peered":"2024-04-25T19:13:04.940638+0000","last_clean":"2024-04-25T19:13:04.940638+0000","last_became_active":"2024-04-25T19:12:44.701778+0000","last_became_peered":"2024-04-25T19:12:44.701778+0000","last_unstale":"2024-04-25T19:13:04.940638+0000","last_undegraded":"2024-04-25T19:13:04.940638+0000","last_fullsized":"2024-04-25T19:13:04.940638+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:36.123098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336490+0000","last_change":"2024-04-25T19:12:43.686223+0000","last_active":"2024-04-25T19:13:05.336490+0000","last_peered":"2024-04-25T19:13:05.336490+0000","last_clean":"2024-04-25T19:13:05.336490+0000","last_became_active":"2024-04-25T19:12:43.685881+0000","last_became_peered":"2024-04-25T19:12:43.685881+0000","last_unstale":"2024-04-25T19:13:05.336490+0000","last_undegraded":"2024-04-25T19:13:05.336490+0000","last_fullsized":"2024-04-25T19:13:05.336490+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:08.114375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.290138+0000","last_change":"2024-04-25T19:13:03.935762+0000","last_active":"2024-04-25T19:13:05.290138+0000","last_peered":"2024-04-25T19:13:05.290138+0000","last_clean":"2024-04-25T19:13:05.290138+0000","last_became_active":"2024-04-25T19:13:03.935475+0000","last_became_peered":"2024-04-25T19:13:03.935475+0000","last_unstale":"2024-04-25T19:13:05.290138+0000","last_undegraded":"2024-04-25T19:13:05.290138+0000","last_fullsized":"2024-04-25T19:13:05.290138+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:12:44.563139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"38'8","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941616+0000","last_change":"2024-04-25T19:12:44.706605+0000","last_active":"2024-04-25T19:13:04.941616+0000","last_peered":"2024-04-25T19:13:04.941616+0000","last_clean":"2024-04-25T19:13:04.941616+0000","last_became_active":"2024-04-25T19:12:44.706192+0000","last_became_peered":"2024-04-25T19:12:44.706192+0000","last_unstale":"2024-04-25T19:13:04.941616+0000","last_undegraded":"2024-04-25T19:13:04.941616+0000","last_fullsized":"2024-04-25T19:13:04.941616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:48:50.429366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940439+0000","last_change":"2024-04-25T19:12:43.689029+0000","last_active":"2024-04-25T19:13:04.940439+0000","last_peered":"2024-04-25T19:13:04.940439+0000","last_clean":"2024-04-25T19:13:04.940439+0000","last_became_active":"2024-04-25T19:12:43.688728+0000","last_became_peered":"2024-04-25T19:12:43.688728+0000","last_unstale":"2024-04-25T19:13:04.940439+0000","last_undegraded":"2024-04-25T19:13:04.940439+0000","last_fullsized":"2024-04-25T19:13:04.940439+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":156,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942115+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:13:04.942115+0000","last_peered":"2024-04-25T19:13:04.942115+0000","last_clean":"2024-04-25T19:13:04.942115+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:13:04.942115+0000","last_undegraded":"2024-04-25T19:13:04.942115+0000","last_fullsized":"2024-04-25T19:13:04.942115+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":6291456,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12582912,"data_stored":12582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":504,"ondisk_log_size":504,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":98,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34332,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337508,"statfs":{"total":96636764160,"available":96601608192,"internally_reserved":0,"allocated":3227648,"data_stored":2931698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":236922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37860,"kb_used_data":6668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333980,"statfs":{"total":96636764160,"available":96597995520,"internally_reserved":0,"allocated":6828032,"data_stored":6535634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":524288,"data_stored":236922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35800,"kb_used_data":4620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336040,"statfs":{"total":96636764160,"available":96600104960,"internally_reserved":0,"allocated":4730880,"data_stored":4438482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":524288,"data_stored":236922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":524288,"data_stored":236922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1130496,"data_stored":834546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:13:08.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:13:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.148+0000 7f9d23f9d640 1 Processor -- start 2024-04-25T19:13:08.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.148+0000 7f9d23f9d640 1 -- start start 2024-04-25T19:13:08.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.149+0000 7f9d23f9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9d1c160e00 0x7f9d1c15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.149+0000 7f9d23f9d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d1c161210 con 0x7f9d1c164990 2024-04-25T19:13:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.149+0000 7f9d23f9d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d1c1614a0 con 0x7f9d1c0b7d70 2024-04-25T19:13:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.149+0000 7f9d23f9d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f9d1c163eb0 con 0x7f9d1c160e00 2024-04-25T19:13:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.149+0000 7f9d21d39640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9d1c164990 0x7f9d1c0b7660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40992/0 (socket says 172.21.15.28:40992) 2024-04-25T19:13:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.149+0000 7f9d21d39640 1 -- 172.21.15.28:0/3545796625 learned_addr learned my addr 172.21.15.28:0/3545796625 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d2253a640 1 --2- 172.21.15.28:0/3545796625 >> v2:172.21.15.53:3300/0 conn(0x7f9d1c160e00 0x7f9d1c15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2039183596 0 0) 0x7f9d1c161210 con 0x7f9d1c164990 2024-04-25T19:13:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d08003520 con 0x7f9d1c164990 2024-04-25T19:13:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1807185625 0 0) 0x7f9d08003520 con 0x7f9d1c164990 2024-04-25T19:13:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 >> v2:172.21.15.53:3300/0 conn(0x7f9d1c160e00 msgr2=0x7f9d1c15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 --2- 172.21.15.28:0/3545796625 >> v2:172.21.15.53:3300/0 conn(0x7f9d1c160e00 0x7f9d1c15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 >> v1:172.21.15.53:6789/0 conn(0x7f9d1c0b7d70 legacy=0x7f9d1c1606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1c164190 con 0x7f9d1c164990 2024-04-25T19:13:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f9d140019f0 con 0x7f9d1c164990 2024-04-25T19:13:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 >> v1:172.21.15.28:6789/0 conn(0x7f9d1c164990 legacy=0x7f9d1c0b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d080039b0 0x7f9d08003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08004800 0x7f9d08004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d08005150 0x7f9d0800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c163eb0 con 0x7f9d08005150 2024-04-25T19:13:08.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9d0800c0c0 con 0x7f9d080039b0 2024-04-25T19:13:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9d0800c2c0 con 0x7f9d08004800 2024-04-25T19:13:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d2253a640 1 --2- 172.21.15.28:0/3545796625 >> v2:172.21.15.53:3300/0 conn(0x7f9d1c160e00 0x7f9d1c15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d21d39640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d080039b0 0x7f9d08003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d2253a640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d08005150 0x7f9d0800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.150+0000 7f9d21538640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08004800 0x7f9d08004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.151+0000 7f9d2253a640 1 -- 172.21.15.28:0/3545796625 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08004800 msgr2=0x7f9d08004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.151+0000 7f9d2253a640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08004800 0x7f9d08004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.151+0000 7f9d2253a640 1 -- 172.21.15.28:0/3545796625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d080039b0 msgr2=0x7f9d08003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.151+0000 7f9d2253a640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d080039b0 0x7f9d08003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.151+0000 7f9d2253a640 1 -- 172.21.15.28:0/3545796625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d08003520 con 0x7f9d08005150 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.151+0000 7f9d2253a640 1 -- 172.21.15.28:0/3545796625 reap_dead start 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.151+0000 7f9d2253a640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d08005150 0x7f9d0800b9c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009c70 tx=0x7f9d08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6b2ebbe5e6662e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18009840 con 0x7f9d08005150 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d18004020 con 0x7f9d08005150 2024-04-25T19:13:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d20d37640 1 -- 172.21.15.28:0/3545796625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18004410 con 0x7f9d08005150 2024-04-25T19:13:08.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d23f9d640 1 -- 172.21.15.28:0/3545796625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d08005150 msgr2=0x7f9d0800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d23f9d640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d08005150 0x7f9d0800b9c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009c70 tx=0x7f9d08005830 comp rx=0 tx=0).stop 2024-04-25T19:13:08.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d23f9d640 1 -- 172.21.15.28:0/3545796625 shutdown_connections 2024-04-25T19:13:08.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d23f9d640 1 --2- 172.21.15.28:0/3545796625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d08005150 0x7f9d0800b9c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d23f9d640 1 -- 172.21.15.28:0/3545796625 >> 172.21.15.28:0/3545796625 conn(0x7f9d1c0d2840 msgr2=0x7f9d1c0d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d23f9d640 1 -- 172.21.15.28:0/3545796625 shutdown_connections 2024-04-25T19:13:08.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.152+0000 7f9d23f9d640 1 -- 172.21.15.28:0/3545796625 wait complete. 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.154+0000 7f9d23f9d640 1 Processor -- start 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.154+0000 7f9d23f9d640 1 -- start start 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d23f9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08005150 0x7f9d1c0cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d23f9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c1f9850 0x7f9d1c1f9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d23f9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d1c0d5be0 0x7f9d1c0d5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d23f9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c15de30 con 0x7f9d1c1f9850 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d23f9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1c15dcb0 con 0x7f9d1c0d5be0 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d23f9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9d1c15dfb0 con 0x7f9d08005150 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c1f9850 0x7f9d1c1f9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c1f9850 0x7f9d1c1f9c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51304/0 (socket says 172.21.15.28:51304) 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 -- 172.21.15.28:0/1535736835 learned_addr learned my addr 172.21.15.28:0/1535736835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d2253a640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d1c0d5be0 0x7f9d1c0d5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21d39640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08005150 0x7f9d1c0cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 -- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08005150 msgr2=0x7f9d1c0cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08005150 0x7f9d1c0cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 -- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d1c0d5be0 msgr2=0x7f9d1c0d5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d1c0d5be0 0x7f9d1c0d5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 -- 172.21.15.28:0/1535736835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1c0d6530 con 0x7f9d1c1f9850 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21d39640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08005150 0x7f9d1c0cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d21538640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c1f9850 0x7f9d1c1f9c60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00f7c0 tx=0x7f9d0c00fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d127fc640 1 -- 172.21.15.28:0/1535736835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00fe70 con 0x7f9d1c1f9850 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d127fc640 1 -- 172.21.15.28:0/1535736835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d0c018070 con 0x7f9d1c1f9850 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d127fc640 1 -- 172.21.15.28:0/1535736835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00d730 con 0x7f9d1c1f9850 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.156+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1c0c9860 con 0x7f9d1c1f9850 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.157+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d1c0c9e20 con 0x7f9d1c1f9850 2024-04-25T19:13:08.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.157+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d1c1fa390 con 0x7f9d1c1f9850 2024-04-25T19:13:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.158+0000 7f9d127fc640 1 -- 172.21.15.28:0/1535736835 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9d0c00d8d0 con 0x7f9d1c1f9850 2024-04-25T19:13:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.159+0000 7f9d127fc640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9d000744b0 0x7f9d000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.159+0000 7f9d127fc640 1 -- 172.21.15.28:0/1535736835 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x7f9d0c095530 con 0x7f9d1c1f9850 2024-04-25T19:13:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.159+0000 7f9d21d39640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9d000744b0 0x7f9d000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.159+0000 7f9d21d39640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9d000744b0 0x7f9d000769c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c0d5bb0 tx=0x7f9d14006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.163+0000 7f9d127fc640 1 -- 172.21.15.28:0/1535736835 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d1c0d6530 con 0x7f9d1c1f9850 2024-04-25T19:13:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.344+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9d1c14cb80 con 0x7f9d000744b0 2024-04-25T19:13:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.352+0000 7f9d127fc640 1 -- 172.21.15.28:0/1535736835 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332528 (secure 0 0 0) 0x7f9d1c14cb80 con 0x7f9d000744b0 2024-04-25T19:13:08.357 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:13:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9d000744b0 msgr2=0x7f9d000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9d000744b0 0x7f9d000769c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c0d5bb0 tx=0x7f9d14006040 comp rx=0 tx=0).stop 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c1f9850 msgr2=0x7f9d1c1f9c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c1f9850 0x7f9d1c1f9c60 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c00f7c0 tx=0x7f9d0c00fc80 comp rx=0 tx=0).stop 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 shutdown_connections 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9d000744b0 0x7f9d000769c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d1c0d5be0 0x7f9d1c0d5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d1c1f9850 0x7f9d1c1f9c60 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 --2- 172.21.15.28:0/1535736835 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9d08005150 0x7f9d1c0cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 >> 172.21.15.28:0/1535736835 conn(0x7f9d1c0d2840 msgr2=0x7f9d1c1606f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 shutdown_connections 2024-04-25T19:13:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.356+0000 7f9d23f9d640 1 -- 172.21.15.28:0/1535736835 wait complete. 2024-04-25T19:13:08.368 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-04-25T19:13:06.857579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6881824,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":67,"num_read_kb":55,"num_write":203,"num_write_kb":7817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":660,"ondisk_log_size":660,"up":258,"acting":258,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":250,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267064,"kb_used_data":17592,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754707656,"statfs":{"total":773094113280,"available":772820639744,"internally_reserved":0,"allocated":18014208,"data_stored":15688048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19545,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.945013"},"pg_stats":[{"pgid":"6.3f","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337205+0000","last_change":"2024-04-25T19:12:44.700653+0000","last_active":"2024-04-25T19:13:05.337205+0000","last_peered":"2024-04-25T19:13:05.337205+0000","last_clean":"2024-04-25T19:13:05.337205+0000","last_became_active":"2024-04-25T19:12:44.700326+0000","last_became_peered":"2024-04-25T19:12:44.700326+0000","last_unstale":"2024-04-25T19:13:05.337205+0000","last_undegraded":"2024-04-25T19:13:05.337205+0000","last_fullsized":"2024-04-25T19:13:05.337205+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:03:34.742465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"38'4","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336811+0000","last_change":"2024-04-25T19:12:44.699053+0000","last_active":"2024-04-25T19:13:05.336811+0000","last_peered":"2024-04-25T19:13:05.336811+0000","last_clean":"2024-04-25T19:13:05.336811+0000","last_became_active":"2024-04-25T19:12:44.698718+0000","last_became_peered":"2024-04-25T19:12:44.698718+0000","last_unstale":"2024-04-25T19:13:05.336811+0000","last_undegraded":"2024-04-25T19:13:05.336811+0000","last_fullsized":"2024-04-25T19:13:05.336811+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:05.953300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289673+0000","last_change":"2024-04-25T19:12:44.692595+0000","last_active":"2024-04-25T19:13:05.289673+0000","last_peered":"2024-04-25T19:13:05.289673+0000","last_clean":"2024-04-25T19:13:05.289673+0000","last_became_active":"2024-04-25T19:12:44.692288+0000","last_became_peered":"2024-04-25T19:12:44.692288+0000","last_unstale":"2024-04-25T19:13:05.289673+0000","last_undegraded":"2024-04-25T19:13:05.289673+0000","last_fullsized":"2024-04-25T19:13:05.289673+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:55:25.129409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"38'8","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336974+0000","last_change":"2024-04-25T19:12:44.691657+0000","last_active":"2024-04-25T19:13:05.336974+0000","last_peered":"2024-04-25T19:13:05.336974+0000","last_clean":"2024-04-25T19:13:05.336974+0000","last_became_active":"2024-04-25T19:12:44.691153+0000","last_became_peered":"2024-04-25T19:12:44.691153+0000","last_unstale":"2024-04-25T19:13:05.336974+0000","last_undegraded":"2024-04-25T19:13:05.336974+0000","last_fullsized":"2024-04-25T19:13:05.336974+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:22.202361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"38'4","reported_seq":31,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337501+0000","last_change":"2024-04-25T19:12:44.685404+0000","last_active":"2024-04-25T19:13:05.337501+0000","last_peered":"2024-04-25T19:13:05.337501+0000","last_clean":"2024-04-25T19:13:05.337501+0000","last_became_active":"2024-04-25T19:12:44.685079+0000","last_became_peered":"2024-04-25T19:12:44.685079+0000","last_unstale":"2024-04-25T19:13:05.337501+0000","last_undegraded":"2024-04-25T19:13:05.337501+0000","last_fullsized":"2024-04-25T19:13:05.337501+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:24.118674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"38'6","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338064+0000","last_change":"2024-04-25T19:12:44.702769+0000","last_active":"2024-04-25T19:13:05.338064+0000","last_peered":"2024-04-25T19:13:05.338064+0000","last_clean":"2024-04-25T19:13:05.338064+0000","last_became_active":"2024-04-25T19:12:44.702437+0000","last_became_peered":"2024-04-25T19:12:44.702437+0000","last_unstale":"2024-04-25T19:13:05.338064+0000","last_undegraded":"2024-04-25T19:13:05.338064+0000","last_fullsized":"2024-04-25T19:13:05.338064+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:22.449658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"38'5","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941003+0000","last_change":"2024-04-25T19:12:44.705815+0000","last_active":"2024-04-25T19:13:04.941003+0000","last_peered":"2024-04-25T19:13:04.941003+0000","last_clean":"2024-04-25T19:13:04.941003+0000","last_became_active":"2024-04-25T19:12:44.705457+0000","last_became_peered":"2024-04-25T19:12:44.705457+0000","last_unstale":"2024-04-25T19:13:04.941003+0000","last_undegraded":"2024-04-25T19:13:04.941003+0000","last_fullsized":"2024-04-25T19:13:04.941003+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:27.321614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336344+0000","last_change":"2024-04-25T19:12:44.701020+0000","last_active":"2024-04-25T19:13:05.336344+0000","last_peered":"2024-04-25T19:13:05.336344+0000","last_clean":"2024-04-25T19:13:05.336344+0000","last_became_active":"2024-04-25T19:12:44.700672+0000","last_became_peered":"2024-04-25T19:12:44.700672+0000","last_unstale":"2024-04-25T19:13:05.336344+0000","last_undegraded":"2024-04-25T19:13:05.336344+0000","last_fullsized":"2024-04-25T19:13:05.336344+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:31.406618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"38'11","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.290055+0000","last_change":"2024-04-25T19:12:44.693607+0000","last_active":"2024-04-25T19:13:05.290055+0000","last_peered":"2024-04-25T19:13:05.290055+0000","last_clean":"2024-04-25T19:13:05.290055+0000","last_became_active":"2024-04-25T19:12:44.693303+0000","last_became_peered":"2024-04-25T19:12:44.693303+0000","last_unstale":"2024-04-25T19:13:05.290055+0000","last_undegraded":"2024-04-25T19:13:05.290055+0000","last_fullsized":"2024-04-25T19:13:05.290055+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:14.908856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"38'15","reported_seq":42,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940826+0000","last_change":"2024-04-25T19:12:44.695584+0000","last_active":"2024-04-25T19:13:04.940826+0000","last_peered":"2024-04-25T19:13:04.940826+0000","last_clean":"2024-04-25T19:13:04.940826+0000","last_became_active":"2024-04-25T19:12:44.695174+0000","last_became_peered":"2024-04-25T19:12:44.695174+0000","last_unstale":"2024-04-25T19:13:04.940826+0000","last_undegraded":"2024-04-25T19:13:04.940826+0000","last_fullsized":"2024-04-25T19:13:04.940826+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:18.897678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"38'6","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941939+0000","last_change":"2024-04-25T19:12:44.692511+0000","last_active":"2024-04-25T19:13:04.941939+0000","last_peered":"2024-04-25T19:13:04.941939+0000","last_clean":"2024-04-25T19:13:04.941939+0000","last_became_active":"2024-04-25T19:12:44.692175+0000","last_became_peered":"2024-04-25T19:12:44.692175+0000","last_unstale":"2024-04-25T19:13:04.941939+0000","last_undegraded":"2024-04-25T19:13:04.941939+0000","last_fullsized":"2024-04-25T19:13:04.941939+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:49.760886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289765+0000","last_change":"2024-04-25T19:12:44.691012+0000","last_active":"2024-04-25T19:13:05.289765+0000","last_peered":"2024-04-25T19:13:05.289765+0000","last_clean":"2024-04-25T19:13:05.289765+0000","last_became_active":"2024-04-25T19:12:44.690627+0000","last_became_peered":"2024-04-25T19:12:44.690627+0000","last_unstale":"2024-04-25T19:13:05.289765+0000","last_undegraded":"2024-04-25T19:13:05.289765+0000","last_fullsized":"2024-04-25T19:13:05.289765+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:21.849594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"38'10","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941342+0000","last_change":"2024-04-25T19:12:44.696238+0000","last_active":"2024-04-25T19:13:04.941342+0000","last_peered":"2024-04-25T19:13:04.941342+0000","last_clean":"2024-04-25T19:13:04.941342+0000","last_became_active":"2024-04-25T19:12:44.695884+0000","last_became_peered":"2024-04-25T19:12:44.695884+0000","last_unstale":"2024-04-25T19:13:04.941342+0000","last_undegraded":"2024-04-25T19:13:04.941342+0000","last_fullsized":"2024-04-25T19:13:04.941342+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:11.387331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942314+0000","last_change":"2024-04-25T19:12:44.690998+0000","last_active":"2024-04-25T19:13:04.942314+0000","last_peered":"2024-04-25T19:13:04.942314+0000","last_clean":"2024-04-25T19:13:04.942314+0000","last_became_active":"2024-04-25T19:12:44.690126+0000","last_became_peered":"2024-04-25T19:12:44.690126+0000","last_unstale":"2024-04-25T19:13:04.942314+0000","last_undegraded":"2024-04-25T19:13:04.942314+0000","last_fullsized":"2024-04-25T19:13:04.942314+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:29.311080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941243+0000","last_change":"2024-04-25T19:12:44.681433+0000","last_active":"2024-04-25T19:13:04.941243+0000","last_peered":"2024-04-25T19:13:04.941243+0000","last_clean":"2024-04-25T19:13:04.941243+0000","last_became_active":"2024-04-25T19:12:44.681100+0000","last_became_peered":"2024-04-25T19:12:44.681100+0000","last_unstale":"2024-04-25T19:13:04.941243+0000","last_undegraded":"2024-04-25T19:13:04.941243+0000","last_fullsized":"2024-04-25T19:13:04.941243+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:03:34.742465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941600+0000","last_change":"2024-04-25T19:12:44.701150+0000","last_active":"2024-04-25T19:13:04.941600+0000","last_peered":"2024-04-25T19:13:04.941600+0000","last_clean":"2024-04-25T19:13:04.941600+0000","last_became_active":"2024-04-25T19:12:44.699787+0000","last_became_peered":"2024-04-25T19:12:44.699787+0000","last_unstale":"2024-04-25T19:13:04.941600+0000","last_undegraded":"2024-04-25T19:13:04.941600+0000","last_fullsized":"2024-04-25T19:13:04.941600+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:24.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940597+0000","last_change":"2024-04-25T19:12:44.706630+0000","last_active":"2024-04-25T19:13:04.940597+0000","last_peered":"2024-04-25T19:13:04.940597+0000","last_clean":"2024-04-25T19:13:04.940597+0000","last_became_active":"2024-04-25T19:12:44.705055+0000","last_became_peered":"2024-04-25T19:12:44.705055+0000","last_unstale":"2024-04-25T19:13:04.940597+0000","last_undegraded":"2024-04-25T19:13:04.940597+0000","last_fullsized":"2024-04-25T19:13:04.940597+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:48:50.429366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2097152,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"38'10","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336993+0000","last_change":"2024-04-25T19:12:44.683787+0000","last_active":"2024-04-25T19:13:05.336993+0000","last_peered":"2024-04-25T19:13:05.336993+0000","last_clean":"2024-04-25T19:13:05.336993+0000","last_became_active":"2024-04-25T19:12:44.683469+0000","last_became_peered":"2024-04-25T19:12:44.683469+0000","last_unstale":"2024-04-25T19:13:05.336993+0000","last_undegraded":"2024-04-25T19:13:05.336993+0000","last_fullsized":"2024-04-25T19:13:05.336993+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:41:54.272820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337488+0000","last_change":"2024-04-25T19:12:44.683518+0000","last_active":"2024-04-25T19:13:05.337488+0000","last_peered":"2024-04-25T19:13:05.337488+0000","last_clean":"2024-04-25T19:13:05.337488+0000","last_became_active":"2024-04-25T19:12:44.683194+0000","last_became_peered":"2024-04-25T19:12:44.683194+0000","last_unstale":"2024-04-25T19:13:05.337488+0000","last_undegraded":"2024-04-25T19:13:05.337488+0000","last_fullsized":"2024-04-25T19:13:05.337488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:37.425520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336950+0000","last_change":"2024-04-25T19:12:44.697957+0000","last_active":"2024-04-25T19:13:05.336950+0000","last_peered":"2024-04-25T19:13:05.336950+0000","last_clean":"2024-04-25T19:13:05.336950+0000","last_became_active":"2024-04-25T19:12:44.697633+0000","last_became_peered":"2024-04-25T19:12:44.697633+0000","last_unstale":"2024-04-25T19:13:05.336950+0000","last_undegraded":"2024-04-25T19:13:05.336950+0000","last_fullsized":"2024-04-25T19:13:05.336950+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:45.995608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"38'11","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941488+0000","last_change":"2024-04-25T19:12:44.690591+0000","last_active":"2024-04-25T19:13:04.941488+0000","last_peered":"2024-04-25T19:13:04.941488+0000","last_clean":"2024-04-25T19:13:04.941488+0000","last_became_active":"2024-04-25T19:12:44.690265+0000","last_became_peered":"2024-04-25T19:12:44.690265+0000","last_unstale":"2024-04-25T19:13:04.941488+0000","last_undegraded":"2024-04-25T19:13:04.941488+0000","last_fullsized":"2024-04-25T19:13:04.941488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:02:57.343073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942727+0000","last_change":"2024-04-25T19:12:44.683217+0000","last_active":"2024-04-25T19:13:04.942727+0000","last_peered":"2024-04-25T19:13:04.942727+0000","last_clean":"2024-04-25T19:13:04.942727+0000","last_became_active":"2024-04-25T19:12:44.682860+0000","last_became_peered":"2024-04-25T19:12:44.682860+0000","last_unstale":"2024-04-25T19:13:04.942727+0000","last_undegraded":"2024-04-25T19:13:04.942727+0000","last_fullsized":"2024-04-25T19:13:04.942727+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:12.662654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940882+0000","last_change":"2024-04-25T19:12:44.704757+0000","last_active":"2024-04-25T19:13:04.940882+0000","last_peered":"2024-04-25T19:13:04.940882+0000","last_clean":"2024-04-25T19:13:04.940882+0000","last_became_active":"2024-04-25T19:12:44.704345+0000","last_became_peered":"2024-04-25T19:12:44.704345+0000","last_unstale":"2024-04-25T19:13:04.940882+0000","last_undegraded":"2024-04-25T19:13:04.940882+0000","last_fullsized":"2024-04-25T19:13:04.940882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:34.452188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"38'6","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336642+0000","last_change":"2024-04-25T19:12:44.685369+0000","last_active":"2024-04-25T19:13:05.336642+0000","last_peered":"2024-04-25T19:13:05.336642+0000","last_clean":"2024-04-25T19:13:05.336642+0000","last_became_active":"2024-04-25T19:12:44.684749+0000","last_became_peered":"2024-04-25T19:12:44.684749+0000","last_unstale":"2024-04-25T19:13:05.336642+0000","last_undegraded":"2024-04-25T19:13:05.336642+0000","last_fullsized":"2024-04-25T19:13:05.336642+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:34.463674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"38'2","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337207+0000","last_change":"2024-04-25T19:12:44.688826+0000","last_active":"2024-04-25T19:13:05.337207+0000","last_peered":"2024-04-25T19:13:05.337207+0000","last_clean":"2024-04-25T19:13:05.337207+0000","last_became_active":"2024-04-25T19:12:44.687859+0000","last_became_peered":"2024-04-25T19:12:44.687859+0000","last_unstale":"2024-04-25T19:13:05.337207+0000","last_undegraded":"2024-04-25T19:13:05.337207+0000","last_fullsized":"2024-04-25T19:13:05.337207+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:13:29.311080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"38'15","reported_seq":44,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338318+0000","last_change":"2024-04-25T19:12:44.698551+0000","last_active":"2024-04-25T19:13:05.338318+0000","last_peered":"2024-04-25T19:13:05.338318+0000","last_clean":"2024-04-25T19:13:05.338318+0000","last_became_active":"2024-04-25T19:12:44.698111+0000","last_became_peered":"2024-04-25T19:12:44.698111+0000","last_unstale":"2024-04-25T19:13:05.338318+0000","last_undegraded":"2024-04-25T19:13:05.338318+0000","last_fullsized":"2024-04-25T19:13:05.338318+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:32:20.222641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337072+0000","last_change":"2024-04-25T19:12:44.681953+0000","last_active":"2024-04-25T19:13:05.337072+0000","last_peered":"2024-04-25T19:13:05.337072+0000","last_clean":"2024-04-25T19:13:05.337072+0000","last_became_active":"2024-04-25T19:12:44.681238+0000","last_became_peered":"2024-04-25T19:12:44.681238+0000","last_unstale":"2024-04-25T19:13:05.337072+0000","last_undegraded":"2024-04-25T19:13:05.337072+0000","last_fullsized":"2024-04-25T19:13:05.337072+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:05.288431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"38'4","reported_seq":31,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941489+0000","last_change":"2024-04-25T19:12:44.706575+0000","last_active":"2024-04-25T19:13:04.941489+0000","last_peered":"2024-04-25T19:13:04.941489+0000","last_clean":"2024-04-25T19:13:04.941489+0000","last_became_active":"2024-04-25T19:12:44.706268+0000","last_became_peered":"2024-04-25T19:12:44.706268+0000","last_unstale":"2024-04-25T19:13:04.941489+0000","last_undegraded":"2024-04-25T19:13:04.941489+0000","last_fullsized":"2024-04-25T19:13:04.941489+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:01.862563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"38'7","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336594+0000","last_change":"2024-04-25T19:12:44.682967+0000","last_active":"2024-04-25T19:13:05.336594+0000","last_peered":"2024-04-25T19:13:05.336594+0000","last_clean":"2024-04-25T19:13:05.336594+0000","last_became_active":"2024-04-25T19:12:44.682477+0000","last_became_peered":"2024-04-25T19:12:44.682477+0000","last_unstale":"2024-04-25T19:13:05.336594+0000","last_undegraded":"2024-04-25T19:13:05.336594+0000","last_fullsized":"2024-04-25T19:13:05.336594+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:28.034410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336708+0000","last_change":"2024-04-25T19:12:44.699586+0000","last_active":"2024-04-25T19:13:05.336708+0000","last_peered":"2024-04-25T19:13:05.336708+0000","last_clean":"2024-04-25T19:13:05.336708+0000","last_became_active":"2024-04-25T19:12:44.699289+0000","last_became_peered":"2024-04-25T19:12:44.699289+0000","last_unstale":"2024-04-25T19:13:05.336708+0000","last_undegraded":"2024-04-25T19:13:05.336708+0000","last_fullsized":"2024-04-25T19:13:05.336708+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:57.582503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"38'12","reported_seq":40,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941024+0000","last_change":"2024-04-25T19:12:44.702160+0000","last_active":"2024-04-25T19:13:04.941024+0000","last_peered":"2024-04-25T19:13:04.941024+0000","last_clean":"2024-04-25T19:13:04.941024+0000","last_became_active":"2024-04-25T19:12:44.701815+0000","last_became_peered":"2024-04-25T19:12:44.701815+0000","last_unstale":"2024-04-25T19:13:04.941024+0000","last_undegraded":"2024-04-25T19:13:04.941024+0000","last_fullsized":"2024-04-25T19:13:04.941024+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:24:14.908856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"38'9","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337918+0000","last_change":"2024-04-25T19:12:44.691655+0000","last_active":"2024-04-25T19:13:05.337918+0000","last_peered":"2024-04-25T19:13:05.337918+0000","last_clean":"2024-04-25T19:13:05.337918+0000","last_became_active":"2024-04-25T19:12:44.691354+0000","last_became_peered":"2024-04-25T19:12:44.691354+0000","last_unstale":"2024-04-25T19:13:05.337918+0000","last_undegraded":"2024-04-25T19:13:05.337918+0000","last_fullsized":"2024-04-25T19:13:05.337918+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:03:12.662654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338522+0000","last_change":"2024-04-25T19:13:03.934638+0000","last_active":"2024-04-25T19:13:05.338522+0000","last_peered":"2024-04-25T19:13:05.338522+0000","last_clean":"2024-04-25T19:13:05.338522+0000","last_became_active":"2024-04-25T19:13:03.934312+0000","last_became_peered":"2024-04-25T19:13:03.934312+0000","last_unstale":"2024-04-25T19:13:05.338522+0000","last_undegraded":"2024-04-25T19:13:05.338522+0000","last_fullsized":"2024-04-25T19:13:05.338522+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:34.745079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940399+0000","last_change":"2024-04-25T19:12:44.705515+0000","last_active":"2024-04-25T19:13:04.940399+0000","last_peered":"2024-04-25T19:13:04.940399+0000","last_clean":"2024-04-25T19:13:04.940399+0000","last_became_active":"2024-04-25T19:12:44.703843+0000","last_became_peered":"2024-04-25T19:12:44.703843+0000","last_unstale":"2024-04-25T19:13:04.940399+0000","last_undegraded":"2024-04-25T19:13:04.940399+0000","last_fullsized":"2024-04-25T19:13:04.940399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:24:52.866281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"42'8","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337346+0000","last_change":"2024-04-25T19:12:43.687421+0000","last_active":"2024-04-25T19:13:05.337346+0000","last_peered":"2024-04-25T19:13:05.337346+0000","last_clean":"2024-04-25T19:13:05.337346+0000","last_became_active":"2024-04-25T19:12:43.687150+0000","last_became_peered":"2024-04-25T19:12:43.687150+0000","last_unstale":"2024-04-25T19:13:05.337346+0000","last_undegraded":"2024-04-25T19:13:05.337346+0000","last_fullsized":"2024-04-25T19:13:05.337346+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:22.014467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":10,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941267+0000","last_change":"2024-04-25T19:13:03.940120+0000","last_active":"2024-04-25T19:13:04.941267+0000","last_peered":"2024-04-25T19:13:04.941267+0000","last_clean":"2024-04-25T19:13:04.941267+0000","last_became_active":"2024-04-25T19:13:03.938104+0000","last_became_peered":"2024-04-25T19:13:03.938104+0000","last_unstale":"2024-04-25T19:13:04.941267+0000","last_undegraded":"2024-04-25T19:13:04.941267+0000","last_fullsized":"2024-04-25T19:13:04.941267+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:16:42.097709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"38'15","reported_seq":44,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338411+0000","last_change":"2024-04-25T19:12:44.698702+0000","last_active":"2024-04-25T19:13:05.338411+0000","last_peered":"2024-04-25T19:13:05.338411+0000","last_clean":"2024-04-25T19:13:05.338411+0000","last_became_active":"2024-04-25T19:12:44.698378+0000","last_became_peered":"2024-04-25T19:12:44.698378+0000","last_unstale":"2024-04-25T19:13:05.338411+0000","last_undegraded":"2024-04-25T19:13:05.338411+0000","last_fullsized":"2024-04-25T19:13:05.338411+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.290035+0000","last_change":"2024-04-25T19:12:43.687002+0000","last_active":"2024-04-25T19:13:05.290035+0000","last_peered":"2024-04-25T19:13:05.290035+0000","last_clean":"2024-04-25T19:13:05.290035+0000","last_became_active":"2024-04-25T19:12:43.686681+0000","last_became_peered":"2024-04-25T19:12:43.686681+0000","last_unstale":"2024-04-25T19:13:05.290035+0000","last_undegraded":"2024-04-25T19:13:05.290035+0000","last_fullsized":"2024-04-25T19:13:05.290035+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337164+0000","last_change":"2024-04-25T19:13:03.942982+0000","last_active":"2024-04-25T19:13:05.337164+0000","last_peered":"2024-04-25T19:13:05.337164+0000","last_clean":"2024-04-25T19:13:05.337164+0000","last_became_active":"2024-04-25T19:13:03.942661+0000","last_became_peered":"2024-04-25T19:13:03.942661+0000","last_unstale":"2024-04-25T19:13:05.337164+0000","last_undegraded":"2024-04-25T19:13:05.337164+0000","last_fullsized":"2024-04-25T19:13:05.337164+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:43.943430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"38'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942242+0000","last_change":"2024-04-25T19:12:44.688116+0000","last_active":"2024-04-25T19:13:04.942242+0000","last_peered":"2024-04-25T19:13:04.942242+0000","last_clean":"2024-04-25T19:13:04.942242+0000","last_became_active":"2024-04-25T19:12:44.687776+0000","last_became_peered":"2024-04-25T19:12:44.687776+0000","last_unstale":"2024-04-25T19:13:04.942242+0000","last_undegraded":"2024-04-25T19:13:04.942242+0000","last_fullsized":"2024-04-25T19:13:04.942242+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:19:45.995608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"42'3","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338591+0000","last_change":"2024-04-25T19:12:43.682264+0000","last_active":"2024-04-25T19:13:05.338591+0000","last_peered":"2024-04-25T19:13:05.338591+0000","last_clean":"2024-04-25T19:13:05.338591+0000","last_became_active":"2024-04-25T19:12:43.681918+0000","last_became_peered":"2024-04-25T19:12:43.681918+0000","last_unstale":"2024-04-25T19:13:05.338591+0000","last_undegraded":"2024-04-25T19:13:05.338591+0000","last_fullsized":"2024-04-25T19:13:05.338591+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:55.257918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338273+0000","last_change":"2024-04-25T19:13:03.938028+0000","last_active":"2024-04-25T19:13:05.338273+0000","last_peered":"2024-04-25T19:13:05.338273+0000","last_clean":"2024-04-25T19:13:05.338273+0000","last_became_active":"2024-04-25T19:13:03.937701+0000","last_became_peered":"2024-04-25T19:13:03.937701+0000","last_unstale":"2024-04-25T19:13:05.338273+0000","last_undegraded":"2024-04-25T19:13:05.338273+0000","last_fullsized":"2024-04-25T19:13:05.338273+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:53.707497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337065+0000","last_change":"2024-04-25T19:12:44.699758+0000","last_active":"2024-04-25T19:13:05.337065+0000","last_peered":"2024-04-25T19:13:05.337065+0000","last_clean":"2024-04-25T19:13:05.337065+0000","last_became_active":"2024-04-25T19:12:44.699294+0000","last_became_peered":"2024-04-25T19:12:44.699294+0000","last_unstale":"2024-04-25T19:13:05.337065+0000","last_undegraded":"2024-04-25T19:13:05.337065+0000","last_fullsized":"2024-04-25T19:13:05.337065+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:48:50.429366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"42'3","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941281+0000","last_change":"2024-04-25T19:12:43.690606+0000","last_active":"2024-04-25T19:13:04.941281+0000","last_peered":"2024-04-25T19:13:04.941281+0000","last_clean":"2024-04-25T19:13:04.941281+0000","last_became_active":"2024-04-25T19:12:43.690163+0000","last_became_peered":"2024-04-25T19:12:43.690163+0000","last_unstale":"2024-04-25T19:13:04.941281+0000","last_undegraded":"2024-04-25T19:13:04.941281+0000","last_fullsized":"2024-04-25T19:13:04.941281+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:53:44.282277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336514+0000","last_change":"2024-04-25T19:13:03.939229+0000","last_active":"2024-04-25T19:13:05.336514+0000","last_peered":"2024-04-25T19:13:05.336514+0000","last_clean":"2024-04-25T19:13:05.336514+0000","last_became_active":"2024-04-25T19:13:03.937472+0000","last_became_peered":"2024-04-25T19:13:03.937472+0000","last_unstale":"2024-04-25T19:13:05.336514+0000","last_undegraded":"2024-04-25T19:13:05.336514+0000","last_fullsized":"2024-04-25T19:13:05.336514+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:36.137436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"38'8","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289514+0000","last_change":"2024-04-25T19:12:44.692946+0000","last_active":"2024-04-25T19:13:05.289514+0000","last_peered":"2024-04-25T19:13:05.289514+0000","last_clean":"2024-04-25T19:13:05.289514+0000","last_became_active":"2024-04-25T19:12:44.692070+0000","last_became_peered":"2024-04-25T19:12:44.692070+0000","last_unstale":"2024-04-25T19:13:05.289514+0000","last_undegraded":"2024-04-25T19:13:05.289514+0000","last_fullsized":"2024-04-25T19:13:05.289514+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:36.123098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942097+0000","last_change":"2024-04-25T19:12:43.682355+0000","last_active":"2024-04-25T19:13:04.942097+0000","last_peered":"2024-04-25T19:13:04.942097+0000","last_clean":"2024-04-25T19:13:04.942097+0000","last_became_active":"2024-04-25T19:12:43.682041+0000","last_became_peered":"2024-04-25T19:12:43.682041+0000","last_unstale":"2024-04-25T19:13:04.942097+0000","last_undegraded":"2024-04-25T19:13:04.942097+0000","last_fullsized":"2024-04-25T19:13:04.942097+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:44.218685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289478+0000","last_change":"2024-04-25T19:13:03.942567+0000","last_active":"2024-04-25T19:13:05.289478+0000","last_peered":"2024-04-25T19:13:05.289478+0000","last_clean":"2024-04-25T19:13:05.289478+0000","last_became_active":"2024-04-25T19:13:03.942266+0000","last_became_peered":"2024-04-25T19:13:03.942266+0000","last_unstale":"2024-04-25T19:13:05.289478+0000","last_undegraded":"2024-04-25T19:13:05.289478+0000","last_fullsized":"2024-04-25T19:13:05.289478+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:09.016195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"38'10","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941388+0000","last_change":"2024-04-25T19:12:44.691213+0000","last_active":"2024-04-25T19:13:04.941388+0000","last_peered":"2024-04-25T19:13:04.941388+0000","last_clean":"2024-04-25T19:13:04.941388+0000","last_became_active":"2024-04-25T19:12:44.690834+0000","last_became_peered":"2024-04-25T19:12:44.690834+0000","last_unstale":"2024-04-25T19:13:04.941388+0000","last_undegraded":"2024-04-25T19:13:04.941388+0000","last_fullsized":"2024-04-25T19:13:04.941388+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:29:16.564095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940110+0000","last_change":"2024-04-25T19:12:43.677641+0000","last_active":"2024-04-25T19:13:04.940110+0000","last_peered":"2024-04-25T19:13:04.940110+0000","last_clean":"2024-04-25T19:13:04.940110+0000","last_became_active":"2024-04-25T19:12:43.677313+0000","last_became_peered":"2024-04-25T19:12:43.677313+0000","last_unstale":"2024-04-25T19:13:04.940110+0000","last_undegraded":"2024-04-25T19:13:04.940110+0000","last_fullsized":"2024-04-25T19:13:04.940110+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:26.370845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289683+0000","last_change":"2024-04-25T19:13:03.937675+0000","last_active":"2024-04-25T19:13:05.289683+0000","last_peered":"2024-04-25T19:13:05.289683+0000","last_clean":"2024-04-25T19:13:05.289683+0000","last_became_active":"2024-04-25T19:13:03.937373+0000","last_became_peered":"2024-04-25T19:13:03.937373+0000","last_unstale":"2024-04-25T19:13:05.289683+0000","last_undegraded":"2024-04-25T19:13:05.289683+0000","last_fullsized":"2024-04-25T19:13:05.289683+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:30.949327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"38'7","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942663+0000","last_change":"2024-04-25T19:12:44.694711+0000","last_active":"2024-04-25T19:13:04.942663+0000","last_peered":"2024-04-25T19:13:04.942663+0000","last_clean":"2024-04-25T19:13:04.942663+0000","last_became_active":"2024-04-25T19:12:44.693493+0000","last_became_peered":"2024-04-25T19:12:44.693493+0000","last_unstale":"2024-04-25T19:13:04.942663+0000","last_undegraded":"2024-04-25T19:13:04.942663+0000","last_fullsized":"2024-04-25T19:13:04.942663+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:18:08.819074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338006+0000","last_change":"2024-04-25T19:12:43.691781+0000","last_active":"2024-04-25T19:13:05.338006+0000","last_peered":"2024-04-25T19:13:05.338006+0000","last_clean":"2024-04-25T19:13:05.338006+0000","last_became_active":"2024-04-25T19:12:43.691468+0000","last_became_peered":"2024-04-25T19:12:43.691468+0000","last_unstale":"2024-04-25T19:13:05.338006+0000","last_undegraded":"2024-04-25T19:13:05.338006+0000","last_fullsized":"2024-04-25T19:13:05.338006+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:49:26.370845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942242+0000","last_change":"2024-04-25T19:13:03.941101+0000","last_active":"2024-04-25T19:13:04.942242+0000","last_peered":"2024-04-25T19:13:04.942242+0000","last_clean":"2024-04-25T19:13:04.942242+0000","last_became_active":"2024-04-25T19:13:03.940729+0000","last_became_peered":"2024-04-25T19:13:03.940729+0000","last_unstale":"2024-04-25T19:13:04.942242+0000","last_undegraded":"2024-04-25T19:13:04.942242+0000","last_fullsized":"2024-04-25T19:13:04.942242+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:55:44.384718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336862+0000","last_change":"2024-04-25T19:12:44.703258+0000","last_active":"2024-04-25T19:13:05.336862+0000","last_peered":"2024-04-25T19:13:05.336862+0000","last_clean":"2024-04-25T19:13:05.336862+0000","last_became_active":"2024-04-25T19:12:44.702920+0000","last_became_peered":"2024-04-25T19:12:44.702920+0000","last_unstale":"2024-04-25T19:13:05.336862+0000","last_undegraded":"2024-04-25T19:13:05.336862+0000","last_fullsized":"2024-04-25T19:13:05.336862+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:29:16.564095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940206+0000","last_change":"2024-04-25T19:12:43.687531+0000","last_active":"2024-04-25T19:13:04.940206+0000","last_peered":"2024-04-25T19:13:04.940206+0000","last_clean":"2024-04-25T19:13:04.940206+0000","last_became_active":"2024-04-25T19:12:43.687230+0000","last_became_peered":"2024-04-25T19:12:43.687230+0000","last_unstale":"2024-04-25T19:13:04.940206+0000","last_undegraded":"2024-04-25T19:13:04.940206+0000","last_fullsized":"2024-04-25T19:13:04.940206+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:56:22.014467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940924+0000","last_change":"2024-04-25T19:13:03.935697+0000","last_active":"2024-04-25T19:13:04.940924+0000","last_peered":"2024-04-25T19:13:04.940924+0000","last_clean":"2024-04-25T19:13:04.940924+0000","last_became_active":"2024-04-25T19:13:03.935360+0000","last_became_peered":"2024-04-25T19:13:03.935360+0000","last_unstale":"2024-04-25T19:13:04.940924+0000","last_undegraded":"2024-04-25T19:13:04.940924+0000","last_fullsized":"2024-04-25T19:13:04.940924+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:29:35.819404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"38'11","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337109+0000","last_change":"2024-04-25T19:12:44.684353+0000","last_active":"2024-04-25T19:13:05.337109+0000","last_peered":"2024-04-25T19:13:05.337109+0000","last_clean":"2024-04-25T19:13:05.337109+0000","last_became_active":"2024-04-25T19:12:44.684024+0000","last_became_peered":"2024-04-25T19:12:44.684024+0000","last_unstale":"2024-04-25T19:13:05.337109+0000","last_undegraded":"2024-04-25T19:13:05.337109+0000","last_fullsized":"2024-04-25T19:13:05.337109+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:45.225366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337340+0000","last_change":"2024-04-25T19:12:43.685597+0000","last_active":"2024-04-25T19:13:05.337340+0000","last_peered":"2024-04-25T19:13:05.337340+0000","last_clean":"2024-04-25T19:13:05.337340+0000","last_became_active":"2024-04-25T19:12:43.685245+0000","last_became_peered":"2024-04-25T19:12:43.685245+0000","last_unstale":"2024-04-25T19:13:05.337340+0000","last_undegraded":"2024-04-25T19:13:05.337340+0000","last_fullsized":"2024-04-25T19:13:05.337340+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:35.845516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337154+0000","last_change":"2024-04-25T19:13:03.935954+0000","last_active":"2024-04-25T19:13:05.337154+0000","last_peered":"2024-04-25T19:13:05.337154+0000","last_clean":"2024-04-25T19:13:05.337154+0000","last_became_active":"2024-04-25T19:13:03.935656+0000","last_became_peered":"2024-04-25T19:13:03.935656+0000","last_unstale":"2024-04-25T19:13:05.337154+0000","last_undegraded":"2024-04-25T19:13:05.337154+0000","last_fullsized":"2024-04-25T19:13:05.337154+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:13:16.514939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941239+0000","last_change":"2024-04-25T19:12:44.698686+0000","last_active":"2024-04-25T19:13:04.941239+0000","last_peered":"2024-04-25T19:13:04.941239+0000","last_clean":"2024-04-25T19:13:04.941239+0000","last_became_active":"2024-04-25T19:12:44.697877+0000","last_became_peered":"2024-04-25T19:12:44.697877+0000","last_unstale":"2024-04-25T19:13:04.941239+0000","last_undegraded":"2024-04-25T19:13:04.941239+0000","last_fullsized":"2024-04-25T19:13:04.941239+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:11:08.672351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941558+0000","last_change":"2024-04-25T19:12:43.691155+0000","last_active":"2024-04-25T19:13:04.941558+0000","last_peered":"2024-04-25T19:13:04.941558+0000","last_clean":"2024-04-25T19:13:04.941558+0000","last_became_active":"2024-04-25T19:12:43.690735+0000","last_became_peered":"2024-04-25T19:12:43.690735+0000","last_unstale":"2024-04-25T19:13:04.941558+0000","last_undegraded":"2024-04-25T19:13:04.941558+0000","last_fullsized":"2024-04-25T19:13:04.941558+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:36.418839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940523+0000","last_change":"2024-04-25T19:13:03.941630+0000","last_active":"2024-04-25T19:13:04.940523+0000","last_peered":"2024-04-25T19:13:04.940523+0000","last_clean":"2024-04-25T19:13:04.940523+0000","last_became_active":"2024-04-25T19:13:03.941304+0000","last_became_peered":"2024-04-25T19:13:03.941304+0000","last_unstale":"2024-04-25T19:13:04.940523+0000","last_undegraded":"2024-04-25T19:13:04.940523+0000","last_fullsized":"2024-04-25T19:13:04.940523+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:12:09.016195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"38'10","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289882+0000","last_change":"2024-04-25T19:12:44.698058+0000","last_active":"2024-04-25T19:13:05.289882+0000","last_peered":"2024-04-25T19:13:05.289882+0000","last_clean":"2024-04-25T19:13:05.289882+0000","last_became_active":"2024-04-25T19:12:44.695384+0000","last_became_peered":"2024-04-25T19:12:44.695384+0000","last_unstale":"2024-04-25T19:13:05.289882+0000","last_undegraded":"2024-04-25T19:13:05.289882+0000","last_fullsized":"2024-04-25T19:13:05.289882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:50:24.688121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"42'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942246+0000","last_change":"2024-04-25T19:12:43.686664+0000","last_active":"2024-04-25T19:13:04.942246+0000","last_peered":"2024-04-25T19:13:04.942246+0000","last_clean":"2024-04-25T19:13:04.942246+0000","last_became_active":"2024-04-25T19:12:43.686375+0000","last_became_peered":"2024-04-25T19:12:43.686375+0000","last_unstale":"2024-04-25T19:13:04.942246+0000","last_undegraded":"2024-04-25T19:13:04.942246+0000","last_fullsized":"2024-04-25T19:13:04.942246+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:35.845516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941920+0000","last_change":"2024-04-25T19:13:03.942015+0000","last_active":"2024-04-25T19:13:04.941920+0000","last_peered":"2024-04-25T19:13:04.941920+0000","last_clean":"2024-04-25T19:13:04.941920+0000","last_became_active":"2024-04-25T19:13:03.941240+0000","last_became_peered":"2024-04-25T19:13:03.941240+0000","last_unstale":"2024-04-25T19:13:04.941920+0000","last_undegraded":"2024-04-25T19:13:04.941920+0000","last_fullsized":"2024-04-25T19:13:04.941920+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:41.104903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941947+0000","last_change":"2024-04-25T19:12:44.690212+0000","last_active":"2024-04-25T19:13:04.941947+0000","last_peered":"2024-04-25T19:13:04.941947+0000","last_clean":"2024-04-25T19:13:04.941947+0000","last_became_active":"2024-04-25T19:12:44.689709+0000","last_became_peered":"2024-04-25T19:12:44.689709+0000","last_unstale":"2024-04-25T19:13:04.941947+0000","last_undegraded":"2024-04-25T19:13:04.941947+0000","last_fullsized":"2024-04-25T19:13:04.941947+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:40.863519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"42'23","reported_seq":52,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337470+0000","last_change":"2024-04-25T19:12:43.689503+0000","last_active":"2024-04-25T19:13:05.337470+0000","last_peered":"2024-04-25T19:13:05.337470+0000","last_clean":"2024-04-25T19:13:05.337470+0000","last_became_active":"2024-04-25T19:12:43.689203+0000","last_became_peered":"2024-04-25T19:12:43.689203+0000","last_unstale":"2024-04-25T19:13:05.337470+0000","last_undegraded":"2024-04-25T19:13:05.337470+0000","last_fullsized":"2024-04-25T19:13:05.337470+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:08:45.991172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941389+0000","last_change":"2024-04-25T19:13:03.939799+0000","last_active":"2024-04-25T19:13:04.941389+0000","last_peered":"2024-04-25T19:13:04.941389+0000","last_clean":"2024-04-25T19:13:04.941389+0000","last_became_active":"2024-04-25T19:13:03.939429+0000","last_became_peered":"2024-04-25T19:13:03.939429+0000","last_unstale":"2024-04-25T19:13:04.941389+0000","last_undegraded":"2024-04-25T19:13:04.941389+0000","last_fullsized":"2024-04-25T19:13:04.941389+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:30.642640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"38'6","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289544+0000","last_change":"2024-04-25T19:12:44.693888+0000","last_active":"2024-04-25T19:13:05.289544+0000","last_peered":"2024-04-25T19:13:05.289544+0000","last_clean":"2024-04-25T19:13:05.289544+0000","last_became_active":"2024-04-25T19:12:44.691696+0000","last_became_peered":"2024-04-25T19:12:44.691696+0000","last_unstale":"2024-04-25T19:13:05.289544+0000","last_undegraded":"2024-04-25T19:13:05.289544+0000","last_fullsized":"2024-04-25T19:13:05.289544+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:24:52.866281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"42'5","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337179+0000","last_change":"2024-04-25T19:12:43.691713+0000","last_active":"2024-04-25T19:13:05.337179+0000","last_peered":"2024-04-25T19:13:05.337179+0000","last_clean":"2024-04-25T19:13:05.337179+0000","last_became_active":"2024-04-25T19:12:43.691375+0000","last_became_peered":"2024-04-25T19:12:43.691375+0000","last_unstale":"2024-04-25T19:13:05.337179+0000","last_undegraded":"2024-04-25T19:13:05.337179+0000","last_fullsized":"2024-04-25T19:13:05.337179+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:23:58.179259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336426+0000","last_change":"2024-04-25T19:13:03.939456+0000","last_active":"2024-04-25T19:13:05.336426+0000","last_peered":"2024-04-25T19:13:05.336426+0000","last_clean":"2024-04-25T19:13:05.336426+0000","last_became_active":"2024-04-25T19:13:03.939169+0000","last_became_peered":"2024-04-25T19:13:03.939169+0000","last_unstale":"2024-04-25T19:13:05.336426+0000","last_undegraded":"2024-04-25T19:13:05.336426+0000","last_fullsized":"2024-04-25T19:13:05.336426+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:16:42.097709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336810+0000","last_change":"2024-04-25T19:12:44.699213+0000","last_active":"2024-04-25T19:13:05.336810+0000","last_peered":"2024-04-25T19:13:05.336810+0000","last_clean":"2024-04-25T19:13:05.336810+0000","last_became_active":"2024-04-25T19:12:44.698893+0000","last_became_peered":"2024-04-25T19:12:44.698893+0000","last_unstale":"2024-04-25T19:13:05.336810+0000","last_undegraded":"2024-04-25T19:13:05.336810+0000","last_fullsized":"2024-04-25T19:13:05.336810+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:09:01.862563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338114+0000","last_change":"2024-04-25T19:12:43.691127+0000","last_active":"2024-04-25T19:13:05.338114+0000","last_peered":"2024-04-25T19:13:05.338114+0000","last_clean":"2024-04-25T19:13:05.338114+0000","last_became_active":"2024-04-25T19:12:43.690372+0000","last_became_peered":"2024-04-25T19:12:43.690372+0000","last_unstale":"2024-04-25T19:13:05.338114+0000","last_undegraded":"2024-04-25T19:13:05.338114+0000","last_fullsized":"2024-04-25T19:13:05.338114+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289962+0000","last_change":"2024-04-25T19:13:03.935754+0000","last_active":"2024-04-25T19:13:05.289962+0000","last_peered":"2024-04-25T19:13:05.289962+0000","last_clean":"2024-04-25T19:13:05.289962+0000","last_became_active":"2024-04-25T19:13:03.935442+0000","last_became_peered":"2024-04-25T19:13:03.935442+0000","last_unstale":"2024-04-25T19:13:05.289962+0000","last_undegraded":"2024-04-25T19:13:05.289962+0000","last_fullsized":"2024-04-25T19:13:05.289962+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:38.152987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"38'10","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940049+0000","last_change":"2024-04-25T19:12:44.698557+0000","last_active":"2024-04-25T19:13:04.940049+0000","last_peered":"2024-04-25T19:13:04.940049+0000","last_clean":"2024-04-25T19:13:04.940049+0000","last_became_active":"2024-04-25T19:12:44.698170+0000","last_became_peered":"2024-04-25T19:12:44.698170+0000","last_unstale":"2024-04-25T19:13:04.940049+0000","last_undegraded":"2024-04-25T19:13:04.940049+0000","last_fullsized":"2024-04-25T19:13:04.940049+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:46:17.053151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336583+0000","last_change":"2024-04-25T19:12:43.687570+0000","last_active":"2024-04-25T19:13:05.336583+0000","last_peered":"2024-04-25T19:13:05.336583+0000","last_clean":"2024-04-25T19:13:05.336583+0000","last_became_active":"2024-04-25T19:12:43.687216+0000","last_became_peered":"2024-04-25T19:12:43.687216+0000","last_unstale":"2024-04-25T19:13:05.336583+0000","last_undegraded":"2024-04-25T19:13:05.336583+0000","last_fullsized":"2024-04-25T19:13:05.336583+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:41:56.285928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336635+0000","last_change":"2024-04-25T19:13:03.938560+0000","last_active":"2024-04-25T19:13:05.336635+0000","last_peered":"2024-04-25T19:13:05.336635+0000","last_clean":"2024-04-25T19:13:05.336635+0000","last_became_active":"2024-04-25T19:13:03.936631+0000","last_became_peered":"2024-04-25T19:13:03.936631+0000","last_unstale":"2024-04-25T19:13:05.336635+0000","last_undegraded":"2024-04-25T19:13:05.336635+0000","last_fullsized":"2024-04-25T19:13:05.336635+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:14:00.227930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"38'15","reported_seq":43,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337001+0000","last_change":"2024-04-25T19:12:44.699754+0000","last_active":"2024-04-25T19:13:05.337001+0000","last_peered":"2024-04-25T19:13:05.337001+0000","last_clean":"2024-04-25T19:13:05.337001+0000","last_became_active":"2024-04-25T19:12:44.699422+0000","last_became_peered":"2024-04-25T19:12:44.699422+0000","last_unstale":"2024-04-25T19:13:05.337001+0000","last_undegraded":"2024-04-25T19:13:05.337001+0000","last_fullsized":"2024-04-25T19:13:05.337001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:11.694018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"42'15","reported_seq":44,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941843+0000","last_change":"2024-04-25T19:12:43.677420+0000","last_active":"2024-04-25T19:13:04.941843+0000","last_peered":"2024-04-25T19:13:04.941843+0000","last_clean":"2024-04-25T19:13:04.941843+0000","last_became_active":"2024-04-25T19:12:43.677108+0000","last_became_peered":"2024-04-25T19:12:43.677108+0000","last_unstale":"2024-04-25T19:13:04.941843+0000","last_undegraded":"2024-04-25T19:13:04.941843+0000","last_fullsized":"2024-04-25T19:13:04.941843+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:32:50.902627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336986+0000","last_change":"2024-04-25T19:13:03.932214+0000","last_active":"2024-04-25T19:13:05.336986+0000","last_peered":"2024-04-25T19:13:05.336986+0000","last_clean":"2024-04-25T19:13:05.336986+0000","last_became_active":"2024-04-25T19:13:03.931746+0000","last_became_peered":"2024-04-25T19:13:03.931746+0000","last_unstale":"2024-04-25T19:13:05.336986+0000","last_undegraded":"2024-04-25T19:13:05.336986+0000","last_fullsized":"2024-04-25T19:13:05.336986+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:55:44.384718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"38'6","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337345+0000","last_change":"2024-04-25T19:12:44.699717+0000","last_active":"2024-04-25T19:13:05.337345+0000","last_peered":"2024-04-25T19:13:05.337345+0000","last_clean":"2024-04-25T19:13:05.337345+0000","last_became_active":"2024-04-25T19:12:44.699361+0000","last_became_peered":"2024-04-25T19:12:44.699361+0000","last_unstale":"2024-04-25T19:13:05.337345+0000","last_undegraded":"2024-04-25T19:13:05.337345+0000","last_fullsized":"2024-04-25T19:13:05.337345+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:04:27.321614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"42'2","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941746+0000","last_change":"2024-04-25T19:12:43.679044+0000","last_active":"2024-04-25T19:13:04.941746+0000","last_peered":"2024-04-25T19:13:04.941746+0000","last_clean":"2024-04-25T19:13:04.941746+0000","last_became_active":"2024-04-25T19:12:43.678723+0000","last_became_peered":"2024-04-25T19:12:43.678723+0000","last_unstale":"2024-04-25T19:13:04.941746+0000","last_undegraded":"2024-04-25T19:13:04.941746+0000","last_fullsized":"2024-04-25T19:13:04.941746+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:15:16.031539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336986+0000","last_change":"2024-04-25T19:13:03.939099+0000","last_active":"2024-04-25T19:13:05.336986+0000","last_peered":"2024-04-25T19:13:05.336986+0000","last_clean":"2024-04-25T19:13:05.336986+0000","last_became_active":"2024-04-25T19:13:03.938624+0000","last_became_peered":"2024-04-25T19:13:03.938624+0000","last_unstale":"2024-04-25T19:13:05.336986+0000","last_undegraded":"2024-04-25T19:13:05.336986+0000","last_fullsized":"2024-04-25T19:13:05.336986+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:13:41.104903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"38'7","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942111+0000","last_change":"2024-04-25T19:12:44.697124+0000","last_active":"2024-04-25T19:13:04.942111+0000","last_peered":"2024-04-25T19:13:04.942111+0000","last_clean":"2024-04-25T19:13:04.942111+0000","last_became_active":"2024-04-25T19:12:44.695351+0000","last_became_peered":"2024-04-25T19:12:44.695351+0000","last_unstale":"2024-04-25T19:13:04.942111+0000","last_undegraded":"2024-04-25T19:13:04.942111+0000","last_fullsized":"2024-04-25T19:13:04.942111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:11.694018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"42'3","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942407+0000","last_change":"2024-04-25T19:12:43.678051+0000","last_active":"2024-04-25T19:13:04.942407+0000","last_peered":"2024-04-25T19:13:04.942407+0000","last_clean":"2024-04-25T19:13:04.942407+0000","last_became_active":"2024-04-25T19:12:43.677640+0000","last_became_peered":"2024-04-25T19:12:43.677640+0000","last_unstale":"2024-04-25T19:13:04.942407+0000","last_undegraded":"2024-04-25T19:13:04.942407+0000","last_fullsized":"2024-04-25T19:13:04.942407+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:55:06.360629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289976+0000","last_change":"2024-04-25T19:13:03.939051+0000","last_active":"2024-04-25T19:13:05.289976+0000","last_peered":"2024-04-25T19:13:05.289976+0000","last_clean":"2024-04-25T19:13:05.289976+0000","last_became_active":"2024-04-25T19:13:03.938713+0000","last_became_peered":"2024-04-25T19:13:03.938713+0000","last_unstale":"2024-04-25T19:13:05.289976+0000","last_undegraded":"2024-04-25T19:13:05.289976+0000","last_fullsized":"2024-04-25T19:13:05.289976+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:03:16.598382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"38'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942178+0000","last_change":"2024-04-25T19:12:44.695748+0000","last_active":"2024-04-25T19:13:04.942178+0000","last_peered":"2024-04-25T19:13:04.942178+0000","last_clean":"2024-04-25T19:13:04.942178+0000","last_became_active":"2024-04-25T19:12:44.695447+0000","last_became_peered":"2024-04-25T19:12:44.695447+0000","last_unstale":"2024-04-25T19:13:04.942178+0000","last_undegraded":"2024-04-25T19:13:04.942178+0000","last_fullsized":"2024-04-25T19:13:04.942178+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:02:05.288431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"42'2","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337501+0000","last_change":"2024-04-25T19:12:43.684026+0000","last_active":"2024-04-25T19:13:05.337501+0000","last_peered":"2024-04-25T19:13:05.337501+0000","last_clean":"2024-04-25T19:13:05.337501+0000","last_became_active":"2024-04-25T19:12:43.683693+0000","last_became_peered":"2024-04-25T19:12:43.683693+0000","last_unstale":"2024-04-25T19:13:05.337501+0000","last_undegraded":"2024-04-25T19:13:05.337501+0000","last_fullsized":"2024-04-25T19:13:05.337501+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:10:14.924183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336996+0000","last_change":"2024-04-25T19:13:03.938890+0000","last_active":"2024-04-25T19:13:05.336996+0000","last_peered":"2024-04-25T19:13:05.336996+0000","last_clean":"2024-04-25T19:13:05.336996+0000","last_became_active":"2024-04-25T19:13:03.938593+0000","last_became_peered":"2024-04-25T19:13:03.938593+0000","last_unstale":"2024-04-25T19:13:05.336996+0000","last_undegraded":"2024-04-25T19:13:05.336996+0000","last_fullsized":"2024-04-25T19:13:05.336996+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:28.376365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"38'9","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337161+0000","last_change":"2024-04-25T19:12:44.701567+0000","last_active":"2024-04-25T19:13:05.337161+0000","last_peered":"2024-04-25T19:13:05.337161+0000","last_clean":"2024-04-25T19:13:05.337161+0000","last_became_active":"2024-04-25T19:12:44.701201+0000","last_became_peered":"2024-04-25T19:12:44.701201+0000","last_unstale":"2024-04-25T19:13:05.337161+0000","last_undegraded":"2024-04-25T19:13:05.337161+0000","last_fullsized":"2024-04-25T19:13:05.337161+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:31:00.405188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":30,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338081+0000","last_change":"2024-04-25T19:12:43.684026+0000","last_active":"2024-04-25T19:13:05.338081+0000","last_peered":"2024-04-25T19:13:05.338081+0000","last_clean":"2024-04-25T19:13:05.338081+0000","last_became_active":"2024-04-25T19:12:43.683700+0000","last_became_peered":"2024-04-25T19:12:43.683700+0000","last_unstale":"2024-04-25T19:13:05.338081+0000","last_undegraded":"2024-04-25T19:13:05.338081+0000","last_fullsized":"2024-04-25T19:13:05.338081+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:33.456993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289574+0000","last_change":"2024-04-25T19:13:03.936847+0000","last_active":"2024-04-25T19:13:05.289574+0000","last_peered":"2024-04-25T19:13:05.289574+0000","last_clean":"2024-04-25T19:13:05.289574+0000","last_became_active":"2024-04-25T19:13:03.936539+0000","last_became_peered":"2024-04-25T19:13:03.936539+0000","last_unstale":"2024-04-25T19:13:05.289574+0000","last_undegraded":"2024-04-25T19:13:05.289574+0000","last_fullsized":"2024-04-25T19:13:05.289574+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:29:35.819404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"38'9","reported_seq":37,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941992+0000","last_change":"2024-04-25T19:12:44.699373+0000","last_active":"2024-04-25T19:13:04.941992+0000","last_peered":"2024-04-25T19:13:04.941992+0000","last_clean":"2024-04-25T19:13:04.941992+0000","last_became_active":"2024-04-25T19:12:44.693763+0000","last_became_peered":"2024-04-25T19:12:44.693763+0000","last_unstale":"2024-04-25T19:13:04.941992+0000","last_undegraded":"2024-04-25T19:13:04.941992+0000","last_fullsized":"2024-04-25T19:13:04.941992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:15.489770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"42'4","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337554+0000","last_change":"2024-04-25T19:12:43.687714+0000","last_active":"2024-04-25T19:13:05.337554+0000","last_peered":"2024-04-25T19:13:05.337554+0000","last_clean":"2024-04-25T19:13:05.337554+0000","last_became_active":"2024-04-25T19:12:43.687375+0000","last_became_peered":"2024-04-25T19:12:43.687375+0000","last_unstale":"2024-04-25T19:13:05.337554+0000","last_undegraded":"2024-04-25T19:13:05.337554+0000","last_fullsized":"2024-04-25T19:13:05.337554+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:15:01.568678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336400+0000","last_change":"2024-04-25T19:13:03.933439+0000","last_active":"2024-04-25T19:13:05.336400+0000","last_peered":"2024-04-25T19:13:05.336400+0000","last_clean":"2024-04-25T19:13:05.336400+0000","last_became_active":"2024-04-25T19:13:03.933097+0000","last_became_peered":"2024-04-25T19:13:03.933097+0000","last_unstale":"2024-04-25T19:13:05.336400+0000","last_undegraded":"2024-04-25T19:13:05.336400+0000","last_fullsized":"2024-04-25T19:13:05.336400+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:24:18.441249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"38'2","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289601+0000","last_change":"2024-04-25T19:12:44.694953+0000","last_active":"2024-04-25T19:13:05.289601+0000","last_peered":"2024-04-25T19:13:05.289601+0000","last_clean":"2024-04-25T19:13:05.289601+0000","last_became_active":"2024-04-25T19:12:44.694607+0000","last_became_peered":"2024-04-25T19:12:44.694607+0000","last_unstale":"2024-04-25T19:13:05.289601+0000","last_undegraded":"2024-04-25T19:13:05.289601+0000","last_fullsized":"2024-04-25T19:13:05.289601+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:34.452188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337309+0000","last_change":"2024-04-25T19:12:43.688266+0000","last_active":"2024-04-25T19:13:05.337309+0000","last_peered":"2024-04-25T19:13:05.337309+0000","last_clean":"2024-04-25T19:13:05.337309+0000","last_became_active":"2024-04-25T19:12:43.687912+0000","last_became_peered":"2024-04-25T19:12:43.687912+0000","last_unstale":"2024-04-25T19:13:05.337309+0000","last_undegraded":"2024-04-25T19:13:05.337309+0000","last_fullsized":"2024-04-25T19:13:05.337309+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:24:54.337918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338296+0000","last_change":"2024-04-25T19:13:03.939466+0000","last_active":"2024-04-25T19:13:05.338296+0000","last_peered":"2024-04-25T19:13:05.338296+0000","last_clean":"2024-04-25T19:13:05.338296+0000","last_became_active":"2024-04-25T19:13:03.939167+0000","last_became_peered":"2024-04-25T19:13:03.939167+0000","last_unstale":"2024-04-25T19:13:05.338296+0000","last_undegraded":"2024-04-25T19:13:05.338296+0000","last_fullsized":"2024-04-25T19:13:05.338296+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:55.378407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"38'5","reported_seq":32,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940795+0000","last_change":"2024-04-25T19:12:44.687324+0000","last_active":"2024-04-25T19:13:04.940795+0000","last_peered":"2024-04-25T19:13:04.940795+0000","last_clean":"2024-04-25T19:13:04.940795+0000","last_became_active":"2024-04-25T19:12:44.686274+0000","last_became_peered":"2024-04-25T19:12:44.686274+0000","last_unstale":"2024-04-25T19:13:04.940795+0000","last_undegraded":"2024-04-25T19:13:04.940795+0000","last_fullsized":"2024-04-25T19:13:04.940795+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:00:05.953300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941165+0000","last_change":"2024-04-25T19:12:43.687265+0000","last_active":"2024-04-25T19:13:04.941165+0000","last_peered":"2024-04-25T19:13:04.941165+0000","last_clean":"2024-04-25T19:13:04.941165+0000","last_became_active":"2024-04-25T19:12:43.686948+0000","last_became_peered":"2024-04-25T19:12:43.686948+0000","last_unstale":"2024-04-25T19:13:04.941165+0000","last_undegraded":"2024-04-25T19:13:04.941165+0000","last_fullsized":"2024-04-25T19:13:04.941165+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:29:55.257918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336484+0000","last_change":"2024-04-25T19:13:03.938813+0000","last_active":"2024-04-25T19:13:05.336484+0000","last_peered":"2024-04-25T19:13:05.336484+0000","last_clean":"2024-04-25T19:13:05.336484+0000","last_became_active":"2024-04-25T19:13:03.938503+0000","last_became_peered":"2024-04-25T19:13:03.938503+0000","last_unstale":"2024-04-25T19:13:05.336484+0000","last_undegraded":"2024-04-25T19:13:05.336484+0000","last_fullsized":"2024-04-25T19:13:05.336484+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:12:19.204323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337126+0000","last_change":"2024-04-25T19:12:44.706030+0000","last_active":"2024-04-25T19:13:05.337126+0000","last_peered":"2024-04-25T19:13:05.337126+0000","last_clean":"2024-04-25T19:13:05.337126+0000","last_became_active":"2024-04-25T19:12:44.704826+0000","last_became_peered":"2024-04-25T19:12:44.704826+0000","last_unstale":"2024-04-25T19:13:05.337126+0000","last_undegraded":"2024-04-25T19:13:05.337126+0000","last_fullsized":"2024-04-25T19:13:05.337126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:11.387331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336728+0000","last_change":"2024-04-25T19:12:43.688066+0000","last_active":"2024-04-25T19:13:05.336728+0000","last_peered":"2024-04-25T19:13:05.336728+0000","last_clean":"2024-04-25T19:13:05.336728+0000","last_became_active":"2024-04-25T19:12:43.687694+0000","last_became_peered":"2024-04-25T19:12:43.687694+0000","last_unstale":"2024-04-25T19:13:05.336728+0000","last_undegraded":"2024-04-25T19:13:05.336728+0000","last_fullsized":"2024-04-25T19:13:05.336728+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:21:24.803044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941093+0000","last_change":"2024-04-25T19:13:03.936444+0000","last_active":"2024-04-25T19:13:04.941093+0000","last_peered":"2024-04-25T19:13:04.941093+0000","last_clean":"2024-04-25T19:13:04.941093+0000","last_became_active":"2024-04-25T19:13:03.936109+0000","last_became_peered":"2024-04-25T19:13:03.936109+0000","last_unstale":"2024-04-25T19:13:04.941093+0000","last_undegraded":"2024-04-25T19:13:04.941093+0000","last_fullsized":"2024-04-25T19:13:04.941093+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:02:30.949327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"38'9","reported_seq":38,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.338016+0000","last_change":"2024-04-25T19:12:44.698861+0000","last_active":"2024-04-25T19:13:05.338016+0000","last_peered":"2024-04-25T19:13:05.338016+0000","last_clean":"2024-04-25T19:13:05.338016+0000","last_became_active":"2024-04-25T19:12:44.698178+0000","last_became_peered":"2024-04-25T19:12:44.698178+0000","last_unstale":"2024-04-25T19:13:05.338016+0000","last_undegraded":"2024-04-25T19:13:05.338016+0000","last_fullsized":"2024-04-25T19:13:05.338016+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:17.267296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"42'3","reported_seq":31,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941845+0000","last_change":"2024-04-25T19:12:43.685799+0000","last_active":"2024-04-25T19:13:04.941845+0000","last_peered":"2024-04-25T19:13:04.941845+0000","last_clean":"2024-04-25T19:13:04.941845+0000","last_became_active":"2024-04-25T19:12:43.685478+0000","last_became_peered":"2024-04-25T19:12:43.685478+0000","last_unstale":"2024-04-25T19:13:04.941845+0000","last_undegraded":"2024-04-25T19:13:04.941845+0000","last_fullsized":"2024-04-25T19:13:04.941845+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:55:21.195681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289842+0000","last_change":"2024-04-25T19:13:03.941282+0000","last_active":"2024-04-25T19:13:05.289842+0000","last_peered":"2024-04-25T19:13:05.289842+0000","last_clean":"2024-04-25T19:13:05.289842+0000","last_became_active":"2024-04-25T19:13:03.940988+0000","last_became_peered":"2024-04-25T19:13:03.940988+0000","last_unstale":"2024-04-25T19:13:05.289842+0000","last_undegraded":"2024-04-25T19:13:05.289842+0000","last_fullsized":"2024-04-25T19:13:05.289842+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:01:30.642640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"38'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941361+0000","last_change":"2024-04-25T19:12:44.699033+0000","last_active":"2024-04-25T19:13:04.941361+0000","last_peered":"2024-04-25T19:13:04.941361+0000","last_clean":"2024-04-25T19:13:04.941361+0000","last_became_active":"2024-04-25T19:12:44.698376+0000","last_became_peered":"2024-04-25T19:12:44.698376+0000","last_unstale":"2024-04-25T19:13:04.941361+0000","last_undegraded":"2024-04-25T19:13:04.941361+0000","last_fullsized":"2024-04-25T19:13:04.941361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:31.406618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"42'5","reported_seq":33,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336945+0000","last_change":"2024-04-25T19:12:43.688186+0000","last_active":"2024-04-25T19:13:05.336945+0000","last_peered":"2024-04-25T19:13:05.336945+0000","last_clean":"2024-04-25T19:13:05.336945+0000","last_became_active":"2024-04-25T19:12:43.687361+0000","last_became_peered":"2024-04-25T19:12:43.687361+0000","last_unstale":"2024-04-25T19:13:05.336945+0000","last_undegraded":"2024-04-25T19:13:05.336945+0000","last_fullsized":"2024-04-25T19:13:05.336945+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:37.742204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337294+0000","last_change":"2024-04-25T19:13:03.942670+0000","last_active":"2024-04-25T19:13:05.337294+0000","last_peered":"2024-04-25T19:13:05.337294+0000","last_clean":"2024-04-25T19:13:05.337294+0000","last_became_active":"2024-04-25T19:13:03.942221+0000","last_became_peered":"2024-04-25T19:13:03.942221+0000","last_unstale":"2024-04-25T19:13:05.337294+0000","last_undegraded":"2024-04-25T19:13:05.337294+0000","last_fullsized":"2024-04-25T19:13:05.337294+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:30:53.707497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"38'7","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336912+0000","last_change":"2024-04-25T19:12:44.705586+0000","last_active":"2024-04-25T19:13:05.336912+0000","last_peered":"2024-04-25T19:13:05.336912+0000","last_clean":"2024-04-25T19:13:05.336912+0000","last_became_active":"2024-04-25T19:12:44.704667+0000","last_became_peered":"2024-04-25T19:12:44.704667+0000","last_unstale":"2024-04-25T19:13:05.336912+0000","last_undegraded":"2024-04-25T19:13:05.336912+0000","last_fullsized":"2024-04-25T19:13:05.336912+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:11:49.760886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289740+0000","last_change":"2024-04-25T19:12:43.685345+0000","last_active":"2024-04-25T19:13:05.289740+0000","last_peered":"2024-04-25T19:13:05.289740+0000","last_clean":"2024-04-25T19:13:05.289740+0000","last_became_active":"2024-04-25T19:12:43.684968+0000","last_became_peered":"2024-04-25T19:12:43.684968+0000","last_unstale":"2024-04-25T19:13:05.289740+0000","last_undegraded":"2024-04-25T19:13:05.289740+0000","last_fullsized":"2024-04-25T19:13:05.289740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T04:42:33.456993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940523+0000","last_change":"2024-04-25T19:13:03.941152+0000","last_active":"2024-04-25T19:13:04.940523+0000","last_peered":"2024-04-25T19:13:04.940523+0000","last_clean":"2024-04-25T19:13:04.940523+0000","last_became_active":"2024-04-25T19:13:03.940693+0000","last_became_peered":"2024-04-25T19:13:03.940693+0000","last_unstale":"2024-04-25T19:13:04.940523+0000","last_undegraded":"2024-04-25T19:13:04.940523+0000","last_fullsized":"2024-04-25T19:13:04.940523+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:14:00.227930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"38'8","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.289674+0000","last_change":"2024-04-25T19:12:44.702919+0000","last_active":"2024-04-25T19:13:05.289674+0000","last_peered":"2024-04-25T19:13:05.289674+0000","last_clean":"2024-04-25T19:13:05.289674+0000","last_became_active":"2024-04-25T19:12:44.702600+0000","last_became_peered":"2024-04-25T19:12:44.702600+0000","last_unstale":"2024-04-25T19:13:05.289674+0000","last_undegraded":"2024-04-25T19:13:05.289674+0000","last_fullsized":"2024-04-25T19:13:05.289674+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:15.489770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":28,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.290049+0000","last_change":"2024-04-25T19:12:43.689072+0000","last_active":"2024-04-25T19:13:05.290049+0000","last_peered":"2024-04-25T19:13:05.290049+0000","last_clean":"2024-04-25T19:13:05.290049+0000","last_became_active":"2024-04-25T19:12:43.688665+0000","last_became_peered":"2024-04-25T19:12:43.688665+0000","last_unstale":"2024-04-25T19:13:05.290049+0000","last_undegraded":"2024-04-25T19:13:05.290049+0000","last_fullsized":"2024-04-25T19:13:05.290049+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:23.111993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940943+0000","last_change":"2024-04-25T19:13:03.937565+0000","last_active":"2024-04-25T19:13:04.940943+0000","last_peered":"2024-04-25T19:13:04.940943+0000","last_clean":"2024-04-25T19:13:04.940943+0000","last_became_active":"2024-04-25T19:13:03.937259+0000","last_became_peered":"2024-04-25T19:13:03.937259+0000","last_unstale":"2024-04-25T19:13:04.940943+0000","last_undegraded":"2024-04-25T19:13:04.940943+0000","last_fullsized":"2024-04-25T19:13:04.940943+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:48:41.764632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"38'5","reported_seq":34,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337977+0000","last_change":"2024-04-25T19:12:44.683442+0000","last_active":"2024-04-25T19:13:05.337977+0000","last_peered":"2024-04-25T19:13:05.337977+0000","last_clean":"2024-04-25T19:13:05.337977+0000","last_became_active":"2024-04-25T19:12:44.683137+0000","last_became_peered":"2024-04-25T19:12:44.683137+0000","last_unstale":"2024-04-25T19:13:05.337977+0000","last_undegraded":"2024-04-25T19:13:05.337977+0000","last_fullsized":"2024-04-25T19:13:05.337977+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:35:16.882127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336777+0000","last_change":"2024-04-25T19:12:43.688566+0000","last_active":"2024-04-25T19:13:05.336777+0000","last_peered":"2024-04-25T19:13:05.336777+0000","last_clean":"2024-04-25T19:13:05.336777+0000","last_became_active":"2024-04-25T19:12:43.688009+0000","last_became_peered":"2024-04-25T19:12:43.688009+0000","last_unstale":"2024-04-25T19:13:05.336777+0000","last_undegraded":"2024-04-25T19:13:05.336777+0000","last_fullsized":"2024-04-25T19:13:05.336777+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:40:47.259986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.337139+0000","last_change":"2024-04-25T19:13:03.938051+0000","last_active":"2024-04-25T19:13:05.337139+0000","last_peered":"2024-04-25T19:13:05.337139+0000","last_clean":"2024-04-25T19:13:05.337139+0000","last_became_active":"2024-04-25T19:13:03.937743+0000","last_became_peered":"2024-04-25T19:13:03.937743+0000","last_unstale":"2024-04-25T19:13:05.337139+0000","last_undegraded":"2024-04-25T19:13:05.337139+0000","last_fullsized":"2024-04-25T19:13:05.337139+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T06:39:15.317830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"38'8","reported_seq":36,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942485+0000","last_change":"2024-04-25T19:12:44.704959+0000","last_active":"2024-04-25T19:13:04.942485+0000","last_peered":"2024-04-25T19:13:04.942485+0000","last_clean":"2024-04-25T19:13:04.942485+0000","last_became_active":"2024-04-25T19:12:44.704200+0000","last_became_peered":"2024-04-25T19:12:44.704200+0000","last_unstale":"2024-04-25T19:13:04.942485+0000","last_undegraded":"2024-04-25T19:13:04.942485+0000","last_fullsized":"2024-04-25T19:13:04.942485+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:23:57.582503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940596+0000","last_change":"2024-04-25T19:12:43.686166+0000","last_active":"2024-04-25T19:13:04.940596+0000","last_peered":"2024-04-25T19:13:04.940596+0000","last_clean":"2024-04-25T19:13:04.940596+0000","last_became_active":"2024-04-25T19:12:43.685724+0000","last_became_peered":"2024-04-25T19:12:43.685724+0000","last_unstale":"2024-04-25T19:13:04.940596+0000","last_undegraded":"2024-04-25T19:13:04.940596+0000","last_fullsized":"2024-04-25T19:13:04.940596+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:00:37.087793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336585+0000","last_change":"2024-04-25T19:13:03.933206+0000","last_active":"2024-04-25T19:13:05.336585+0000","last_peered":"2024-04-25T19:13:05.336585+0000","last_clean":"2024-04-25T19:13:05.336585+0000","last_became_active":"2024-04-25T19:13:03.932874+0000","last_became_peered":"2024-04-25T19:13:03.932874+0000","last_unstale":"2024-04-25T19:13:05.336585+0000","last_undegraded":"2024-04-25T19:13:05.336585+0000","last_fullsized":"2024-04-25T19:13:05.336585+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:08:18.325268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"38'7","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940638+0000","last_change":"2024-04-25T19:12:44.702272+0000","last_active":"2024-04-25T19:13:04.940638+0000","last_peered":"2024-04-25T19:13:04.940638+0000","last_clean":"2024-04-25T19:13:04.940638+0000","last_became_active":"2024-04-25T19:12:44.701778+0000","last_became_peered":"2024-04-25T19:12:44.701778+0000","last_unstale":"2024-04-25T19:13:04.940638+0000","last_undegraded":"2024-04-25T19:13:04.940638+0000","last_fullsized":"2024-04-25T19:13:04.940638+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:06:36.123098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.336490+0000","last_change":"2024-04-25T19:12:43.686223+0000","last_active":"2024-04-25T19:13:05.336490+0000","last_peered":"2024-04-25T19:13:05.336490+0000","last_clean":"2024-04-25T19:13:05.336490+0000","last_became_active":"2024-04-25T19:12:43.685881+0000","last_became_peered":"2024-04-25T19:12:43.685881+0000","last_unstale":"2024-04-25T19:13:05.336490+0000","last_undegraded":"2024-04-25T19:13:05.336490+0000","last_fullsized":"2024-04-25T19:13:05.336490+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:08:08.114375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:05.290138+0000","last_change":"2024-04-25T19:13:03.935762+0000","last_active":"2024-04-25T19:13:05.290138+0000","last_peered":"2024-04-25T19:13:05.290138+0000","last_clean":"2024-04-25T19:13:05.290138+0000","last_became_active":"2024-04-25T19:13:03.935475+0000","last_became_peered":"2024-04-25T19:13:03.935475+0000","last_unstale":"2024-04-25T19:13:05.290138+0000","last_undegraded":"2024-04-25T19:13:05.290138+0000","last_fullsized":"2024-04-25T19:13:05.290138+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:13:02.917314+0000","last_clean_scrub_stamp":"2024-04-25T19:13:02.917314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T07:12:44.563139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"38'8","reported_seq":35,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.941616+0000","last_change":"2024-04-25T19:12:44.706605+0000","last_active":"2024-04-25T19:13:04.941616+0000","last_peered":"2024-04-25T19:13:04.941616+0000","last_clean":"2024-04-25T19:13:04.941616+0000","last_became_active":"2024-04-25T19:12:44.706192+0000","last_became_peered":"2024-04-25T19:12:44.706192+0000","last_unstale":"2024-04-25T19:13:04.941616+0000","last_undegraded":"2024-04-25T19:13:04.941616+0000","last_fullsized":"2024-04-25T19:13:04.941616+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:43.662005+0000","last_clean_scrub_stamp":"2024-04-25T19:12:43.662005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:48:50.429366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":29,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.940439+0000","last_change":"2024-04-25T19:12:43.689029+0000","last_active":"2024-04-25T19:13:04.940439+0000","last_peered":"2024-04-25T19:13:04.940439+0000","last_clean":"2024-04-25T19:13:04.940439+0000","last_became_active":"2024-04-25T19:12:43.688728+0000","last_became_peered":"2024-04-25T19:12:43.688728+0000","last_unstale":"2024-04-25T19:13:04.940439+0000","last_undegraded":"2024-04-25T19:13:04.940439+0000","last_fullsized":"2024-04-25T19:13:04.940439+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:12:42.655324+0000","last_clean_scrub_stamp":"2024-04-25T19:12:42.655324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T05:23:53.649033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":156,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-25T19:13:04.942115+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:13:04.942115+0000","last_peered":"2024-04-25T19:13:04.942115+0000","last_clean":"2024-04-25T19:13:04.942115+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:13:04.942115+0000","last_undegraded":"2024-04-25T19:13:04.942115+0000","last_fullsized":"2024-04-25T19:13:04.942115+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":6291456,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12582912,"data_stored":12582912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":504,"ondisk_log_size":504,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":98,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34332,"kb_used_data":3152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337508,"statfs":{"total":96636764160,"available":96601608192,"internally_reserved":0,"allocated":3227648,"data_stored":2931698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31700,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":524288,"data_stored":236922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37860,"kb_used_data":6668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333980,"statfs":{"total":96636764160,"available":96597995520,"internally_reserved":0,"allocated":6828032,"data_stored":6535634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":524288,"data_stored":236922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574873,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35800,"kb_used_data":4620,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336040,"statfs":{"total":96636764160,"available":96600104960,"internally_reserved":0,"allocated":4730880,"data_stored":4438482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":524288,"data_stored":236922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31696,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":524288,"data_stored":236922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574873,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32284,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339556,"statfs":{"total":96636764160,"available":96603705344,"internally_reserved":0,"allocated":1130496,"data_stored":834546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6291456,"data_stored":6291456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2097152,"data_stored":2097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:13:08.373 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T19:13:08.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -s 2024-04-25T19:13:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.516+0000 7fe18f984640 1 Processor -- start 2024-04-25T19:13:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.516+0000 7fe18f984640 1 -- start start 2024-04-25T19:13:08.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18f984640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe18809b540 0x7fe18809b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18f984640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe18806d830 0x7fe18806dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18f984640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1880cfdb0 con 0x7fe18815adf0 2024-04-25T19:13:08.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18f984640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fe1880d0390 con 0x7fe18806d830 2024-04-25T19:13:08.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18f984640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fe18809be90 con 0x7fe18809b540 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18df21640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe18806d830 0x7fe18806dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18d720640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe18815adf0 0x7fe18815e170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41000/0 (socket says 172.21.15.28:41000) 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18df21640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe18806d830 0x7fe18806dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51314/0 (socket says 172.21.15.28:51314) 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18d720640 1 -- 172.21.15.28:0/1768427094 learned_addr learned my addr 172.21.15.28:0/1768427094 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18cf1f640 1 --2- 172.21.15.28:0/1768427094 >> v2:172.21.15.53:3300/0 conn(0x7fe18809b540 0x7fe18809b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18df21640 1 -- 172.21.15.28:0/1768427094 >> v2:172.21.15.53:3300/0 conn(0x7fe18809b540 msgr2=0x7fe18809b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18df21640 1 --2- 172.21.15.28:0/1768427094 >> v2:172.21.15.53:3300/0 conn(0x7fe18809b540 0x7fe18809b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18df21640 1 -- 172.21.15.28:0/1768427094 >> v1:172.21.15.28:6789/0 conn(0x7fe18815adf0 legacy=0x7fe18815e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe187fff640 1 -- 172.21.15.28:0/1768427094 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2632718281 0 0) 0x7fe1880cfdb0 con 0x7fe18815adf0 2024-04-25T19:13:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.518+0000 7fe18df21640 1 -- 172.21.15.28:0/1768427094 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1880ceeb0 con 0x7fe18806d830 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.519+0000 7fe18df21640 1 --2- 172.21.15.28:0/1768427094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe18806dc40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe1880d3960 tx=0x7fe17802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdc8962a810f28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.519+0000 7fe187fff640 1 -- 172.21.15.28:0/1768427094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe178007740 con 0x7fe18806d830 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.519+0000 7fe187fff640 1 -- 172.21.15.28:0/1768427094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1780078e0 con 0x7fe18806d830 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.519+0000 7fe187fff640 1 -- 172.21.15.28:0/1768427094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe178007cb0 con 0x7fe18806d830 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.520+0000 7fe18f984640 1 -- 172.21.15.28:0/1768427094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 msgr2=0x7fe18806dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.520+0000 7fe18f984640 1 --2- 172.21.15.28:0/1768427094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe18806dc40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe1880d3960 tx=0x7fe17802a680 comp rx=0 tx=0).stop 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.520+0000 7fe18f984640 1 -- 172.21.15.28:0/1768427094 shutdown_connections 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.520+0000 7fe18f984640 1 --2- 172.21.15.28:0/1768427094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe18806dc40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.520+0000 7fe18f984640 1 --2- 172.21.15.28:0/1768427094 >> v2:172.21.15.53:3300/0 conn(0x7fe18809b540 0x7fe18809b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.520+0000 7fe18f984640 1 -- 172.21.15.28:0/1768427094 >> 172.21.15.28:0/1768427094 conn(0x7fe1880b7620 msgr2=0x7fe1880b7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.520+0000 7fe18f984640 1 -- 172.21.15.28:0/1768427094 shutdown_connections 2024-04-25T19:13:08.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.520+0000 7fe18f984640 1 -- 172.21.15.28:0/1768427094 wait complete. 2024-04-25T19:13:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.522+0000 7fe18f984640 1 Processor -- start 2024-04-25T19:13:08.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.522+0000 7fe18f984640 1 -- start start 2024-04-25T19:13:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18f984640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe1881494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18f984640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe18809b540 0x7fe1880d2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18f984640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe18815adf0 0x7fe1881566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18f984640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1880ceb80 con 0x7fe18806d830 2024-04-25T19:13:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18f984640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe1880d0020 con 0x7fe18815adf0 2024-04-25T19:13:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18f984640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe1880cf190 con 0x7fe18809b540 2024-04-25T19:13:08.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18d720640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe1881494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18d720640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe1881494f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51316/0 (socket says 172.21.15.28:51316) 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18d720640 1 -- 172.21.15.28:0/1206503144 learned_addr learned my addr 172.21.15.28:0/1206503144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18df21640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe18815adf0 0x7fe1881566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.523+0000 7fe18cf1f640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe18809b540 0x7fe1880d2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe18d720640 1 -- 172.21.15.28:0/1206503144 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe18809b540 msgr2=0x7fe1880d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe18d720640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe18809b540 0x7fe1880d2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe18d720640 1 -- 172.21.15.28:0/1206503144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe18815adf0 msgr2=0x7fe1881566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe18d720640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe18815adf0 0x7fe1881566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe18d720640 1 -- 172.21.15.28:0/1206503144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe188149230 con 0x7fe18806d830 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe18d720640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe1881494f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe18000c410 tx=0x7fe18000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe185ffb640 1 -- 172.21.15.28:0/1206503144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1800074e0 con 0x7fe18806d830 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe185ffb640 1 -- 172.21.15.28:0/1206503144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1800047e0 con 0x7fe18806d830 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18806cb50 con 0x7fe18806d830 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe185ffb640 1 -- 172.21.15.28:0/1206503144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180004bd0 con 0x7fe18806d830 2024-04-25T19:13:08.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.524+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe188155140 con 0x7fe18806d830 2024-04-25T19:13:08.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.525+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe188050740 con 0x7fe18806d830 2024-04-25T19:13:08.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.526+0000 7fe185ffb640 1 -- 172.21.15.28:0/1206503144 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe180011070 con 0x7fe18806d830 2024-04-25T19:13:08.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.526+0000 7fe185ffb640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1740744b0 0x7fe1740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.526+0000 7fe185ffb640 1 -- 172.21.15.28:0/1206503144 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x7fe1800951a0 con 0x7fe18806d830 2024-04-25T19:13:08.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.526+0000 7fe18cf1f640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1740744b0 0x7fe1740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.527+0000 7fe18cf1f640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1740744b0 0x7fe1740769c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006fd0 tx=0x7fe17c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.531+0000 7fe185ffb640 1 -- 172.21.15.28:0/1206503144 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe18005f5a0 con 0x7fe18806d830 2024-04-25T19:13:08.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.791+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fe1880cf5e0 con 0x7fe18806d830 2024-04-25T19:13:08.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.791+0000 7fe185ffb640 1 -- 172.21.15.28:0/1206503144 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+601 (secure 0 0 0) 0x7fe18005eea0 con 0x7fe18806d830 2024-04-25T19:13:08.793 INFO:teuthology.orchestra.run.smithi028.stdout: cluster: 2024-04-25T19:13:08.793 INFO:teuthology.orchestra.run.smithi028.stdout: id: 5d8007d4-24a5-41be-aa4d-cd64fedac62a 2024-04-25T19:13:08.793 INFO:teuthology.orchestra.run.smithi028.stdout: health: HEALTH_ERR 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: 2 filesystems are degraded 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: 2 filesystems are offline 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: services: 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: mgr: y(active, since 119s), standbys: x 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: mds: 0/2 daemons up (2 failed), 4 standby 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: osd: 8 osds: 8 up (since 113s), 8 in (since 2m) 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: data: 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: volumes: 0/2 healthy, 2 failed 2024-04-25T19:13:08.794 INFO:teuthology.orchestra.run.smithi028.stdout: pools: 4 pools, 129 pgs 2024-04-25T19:13:08.795 INFO:teuthology.orchestra.run.smithi028.stdout: objects: 4 objects, 6.6 MiB 2024-04-25T19:13:08.795 INFO:teuthology.orchestra.run.smithi028.stdout: usage: 261 MiB used, 720 GiB / 720 GiB avail 2024-04-25T19:13:08.795 INFO:teuthology.orchestra.run.smithi028.stdout: pgs: 129 active+clean 2024-04-25T19:13:08.795 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:08.795 INFO:teuthology.orchestra.run.smithi028.stdout: io: 2024-04-25T19:13:08.795 INFO:teuthology.orchestra.run.smithi028.stdout: client: 0 B/s wr, 0 op/s rd, 4 op/s wr 2024-04-25T19:13:08.795 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1740744b0 msgr2=0x7fe1740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1740744b0 0x7fe1740769c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe17c006fd0 tx=0x7fe17c008040 comp rx=0 tx=0).stop 2024-04-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 msgr2=0x7fe1881494f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe1881494f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe18000c410 tx=0x7fe18000c8d0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 shutdown_connections 2024-04-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe1740744b0 0x7fe1740769c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe18815adf0 0x7fe1881566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe18809b540 0x7fe1880d2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 --2- 172.21.15.28:0/1206503144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe18806d830 0x7fe1881494f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 >> 172.21.15.28:0/1206503144 conn(0x7fe1880b7620 msgr2=0x7fe188160ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 shutdown_connections 2024-04-25T19:13:08.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.794+0000 7fe18f984640 1 -- 172.21.15.28:0/1206503144 wait complete. 2024-04-25T19:13:08.807 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 init --force-init --filesystem recovery_fs --alternate-pool recovery_fs_metadata 2024-04-25T19:13:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.860+0000 7f07ce810740 1 -- start start 2024-04-25T19:13:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.860+0000 7f07ce810740 1 Processor -- start 2024-04-25T19:13:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07ce810740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9d680 0x560d38cb6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07ce810740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9db00 0x560d38cb6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07ce810740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9df80 0x560d38cb6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07ce810740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d38cc2600 con 0x560d38d9d680 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07ce810740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560d38cc2480 con 0x560d38d9df80 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07ce810740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x560d38cc2300 con 0x560d38d9db00 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cc634640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9d680 0x560d38cb6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cc634640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9d680 0x560d38cb6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51334/0 (socket says 172.21.15.28:51334) 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cc634640 1 -- 172.21.15.28:0/610544592 learned_addr learned my addr 172.21.15.28:0/610544592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cbe33640 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9db00 0x560d38cb6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cce35640 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9df80 0x560d38cb6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cc634640 1 -- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9db00 msgr2=0x560d38cb6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cc634640 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9db00 0x560d38cb6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cc634640 1 -- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9df80 msgr2=0x560d38cb6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cc634640 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9df80 0x560d38cb6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cc634640 1 -- 172.21.15.28:0/610544592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d34c01680 con 0x560d38d9d680 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.861+0000 7f07cbe33640 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9db00 0x560d38cb6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07cc634640 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9d680 0x560d38cb6000 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x560d39a21c80 tx=0x560d38d08120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59eb84d0030c36f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07c9e2f640 1 -- 172.21.15.28:0/610544592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560d34bfa680 con 0x560d38d9d680 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07c9e2f640 1 -- 172.21.15.28:0/610544592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x560d38c4c1c0 con 0x560d38d9d680 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07c9e2f640 1 -- 172.21.15.28:0/610544592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560d34bfab60 con 0x560d38d9d680 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07ce810740 1 -- 172.21.15.28:0/610544592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d34c01860 con 0x560d38d9d680 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07ce810740 1 -- 172.21.15.28:0/610544592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x560d34c012c0 con 0x560d38d9d680 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07ce810740 4 waiting for MDS map... 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07c9e2f640 1 -- 172.21.15.28:0/610544592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x560d39ae4a00 con 0x560d38d9d680 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07c9e2f640 1 -- 172.21.15.28:0/610544592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x560d38c5d180 con 0x560d38d9d680 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07ce810740 4 Got MDS map 25 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.862+0000 7f07ce810740 4 datascan.main: Using metadata pool output 2024-04-25T19:13:08.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.863+0000 7f07ce810740 4 datascan.main: connecting to RADOS... 2024-04-25T19:13:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.863+0000 7f07ce810740 1 Processor -- start 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.863+0000 7f07ce810740 1 -- start start 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.864+0000 7f07ce810740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.864+0000 7f07ce810740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9c900 0x560d38cb7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.864+0000 7f07ce810740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9e880 0x560d38cb7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.864+0000 7f07ce810740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d38cc2180 con 0x560d38d9e880 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.864+0000 7f07ce810740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560d38c64480 con 0x560d38d9cd80 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.864+0000 7f07ce810740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x560d38c64900 con 0x560d38d9c900 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.864+0000 7f07cbe33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9e880 0x560d38cb7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cbe33640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9e880 0x560d38cb7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51338/0 (socket says 172.21.15.28:51338) 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cbe33640 1 -- 172.21.15.28:0/3476696950 learned_addr learned my addr 172.21.15.28:0/3476696950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cce35640 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cc634640 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9c900 0x560d38cb7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cbe33640 1 -- 172.21.15.28:0/3476696950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9c900 msgr2=0x560d38cb7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cbe33640 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9c900 0x560d38cb7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cbe33640 1 -- 172.21.15.28:0/3476696950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9cd80 msgr2=0x560d38cb7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cbe33640 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cbe33640 1 -- 172.21.15.28:0/3476696950 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d34c014a0 con 0x560d38d9e880 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07cbe33640 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9e880 0x560d38cb7b80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x560d39adaff0 tx=0x560d38d04ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74adfaa3418dddb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07c7e2b640 1 -- 172.21.15.28:0/3476696950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560d34bfa9c0 con 0x560d38d9e880 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07c7e2b640 1 -- 172.21.15.28:0/3476696950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x560d38c4ca80 con 0x560d38d9e880 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.865+0000 7f07c7e2b640 1 -- 172.21.15.28:0/3476696950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560d34bfa680 con 0x560d38d9e880 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 -- 172.21.15.28:0/3476696950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9e880 msgr2=0x560d38cb7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9e880 0x560d38cb7b80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x560d39adaff0 tx=0x560d38d04ae0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 -- 172.21.15.28:0/3476696950 shutdown_connections 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9e880 0x560d38cb7b80 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9c900 0x560d38cb7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 --2- 172.21.15.28:0/3476696950 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 -- 172.21.15.28:0/3476696950 >> 172.21.15.28:0/3476696950 conn(0x560d38d9d200 msgr2=0x560d38ce8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 -- 172.21.15.28:0/3476696950 shutdown_connections 2024-04-25T19:13:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.866+0000 7f07ce810740 1 -- 172.21.15.28:0/3476696950 wait complete. 2024-04-25T19:13:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.867+0000 7f07ce810740 1 Processor -- start 2024-04-25T19:13:08.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.867+0000 7f07ce810740 1 -- start start 2024-04-25T19:13:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07ce810740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9c900 0x560d38cb7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07ce810740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07ce810740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9e880 0x560d38cb7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07ce810740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d38cc3980 con 0x560d38d9cd80 2024-04-25T19:13:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07ce810740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x560d38cc2180 con 0x560d38d9c900 2024-04-25T19:13:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07ce810740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x560d38cc3800 con 0x560d38d9e880 2024-04-25T19:13:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cc634640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cc634640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51340/0 (socket says 172.21.15.28:51340) 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cc634640 1 -- 172.21.15.28:0/3795850831 learned_addr learned my addr 172.21.15.28:0/3795850831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cbe33640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9e880 0x560d38cb7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cce35640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9c900 0x560d38cb7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cc634640 1 -- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9e880 msgr2=0x560d38cb7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cc634640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9e880 0x560d38cb7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cc634640 1 -- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9c900 msgr2=0x560d38cb7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cc634640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9c900 0x560d38cb7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.868+0000 7f07cc634640 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d34c01680 con 0x560d38d9cd80 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.869+0000 7f07cc634640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x560d39adbcb0 tx=0x560d38f16f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.869+0000 7f07c6628640 1 -- 172.21.15.28:0/3795850831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560d34bfa820 con 0x560d38d9cd80 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.869+0000 7f07c6628640 1 -- 172.21.15.28:0/3795850831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x560d38c4d500 con 0x560d38d9cd80 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.869+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x560d34c01860 con 0x560d38d9cd80 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.869+0000 7f07c6628640 1 -- 172.21.15.28:0/3795850831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x560d34bfa9c0 con 0x560d38d9cd80 2024-04-25T19:13:08.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.869+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d34c010e0 con 0x560d38d9cd80 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.869+0000 7f07ce810740 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.870+0000 7f07c6628640 1 -- 172.21.15.28:0/3795850831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x560d39b30900 con 0x560d38d9cd80 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.870+0000 7f07c6628640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x560d38d9fa80 0x560d38cb8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.871+0000 7f07c6628640 1 -- 172.21.15.28:0/3795850831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x560d39b48600 con 0x560d38d9cd80 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.871+0000 7f07cce35640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x560d38d9fa80 0x560d38cb8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.871+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x560d38d9ed00 0x560d38cb8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.871+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:1 7.14 7:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x560d399fd000 con 0x560d38d9ed00 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.871+0000 7f07cbe33640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x560d38d9ed00 0x560d38cb8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.871+0000 7f07cce35640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x560d38d9fa80 0x560d38cb8100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x560d38d07170 tx=0x560d38f88840 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.871+0000 7f07cbe33640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x560d38d9ed00 0x560d38cb8680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.871+0000 7f07cbe33640 1 -- 172.21.15.28:0/3795850831 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== osd_op_reply(1 1.00000000.inode [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 160+0+0 (crc 0 0 0) 0x560d38cec000 con 0x560d38d9ed00 2024-04-25T19:13:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.872+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:2 7.14 7:287034c4:::1.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x560d399fd400 con 0x560d38d9ed00 2024-04-25T19:13:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.873+0000 7f07cbe33640 1 -- 172.21.15.28:0/3795850831 <== osd.3 v2:172.21.15.28:6818/1928233777 2 ==== osd_op_reply(2 1.00000000.inode [writefull 0~582] v46'1 uv1 ondisk = 0) ==== 160+0+0 (crc 0 0 0) 0x560d38cec000 con 0x560d38d9ed00 2024-04-25T19:13:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.873+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x560d38d9f180 0x560d38cb8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.873+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:3 7.1f 7:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x560d39008000 con 0x560d38d9f180 2024-04-25T19:13:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.874+0000 7f07cce35640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x560d38d9f180 0x560d38cb8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.874+0000 7f07cce35640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x560d38d9f180 0x560d38cb8c00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.874+0000 7f07cce35640 1 -- 172.21.15.28:0/3795850831 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== osd_op_reply(3 100.00000000.inode [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 162+0+0 (crc 0 0 0) 0x560d38cec500 con 0x560d38d9f180 2024-04-25T19:13:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.874+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:4 7.1f 7:fe07bba1:::100.00000000.inode:head [writefull 0~582 in=582b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x560d39008400 con 0x560d38d9f180 2024-04-25T19:13:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.876+0000 7f07cce35640 1 -- 172.21.15.28:0/3795850831 <== osd.6 v2:172.21.15.53:6809/3653668189 2 ==== osd_op_reply(4 100.00000000.inode [writefull 0~582] v46'1 uv1 ondisk = 0) ==== 162+0+0 (crc 0 0 0) 0x560d38cec500 con 0x560d38d9f180 2024-04-25T19:13:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.876+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x560d38d9f600 0x560d38cb9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.876+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:5 7.13 7:cd5a64a3:::100.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x560d39008800 con 0x560d38d9f600 2024-04-25T19:13:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.877+0000 7f07cc634640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x560d38d9f600 0x560d38cb9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.877+0000 7f07cc634640 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x560d38d9f600 0x560d38cb9180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.877+0000 7f07cc634640 1 -- 172.21.15.28:0/3795850831 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== osd_op_reply(5 100.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x560d38cec780 con 0x560d38d9f600 2024-04-25T19:13:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.877+0000 7f07ce810740 10 datascan.find_or_create_dirfrag: read_version = 0 2024-04-25T19:13:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.877+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:6 7.13 7:cd5a64a3:::100.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x560d39008c00 con 0x560d38d9f600 2024-04-25T19:13:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.878+0000 7f07cc634640 1 -- 172.21.15.28:0/3795850831 <== osd.0 v2:172.21.15.28:6800/1137456103 2 ==== osd_op_reply(6 100.00000000 [create,omap-set-header] v46'1 uv1 ondisk = 0) ==== 198+0+0 (crc 0 0 0) 0x560d38cec780 con 0x560d38d9f600 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.878+0000 7f07ce810740 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x100 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x560d38d9f600 msgr2=0x560d38cb9180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x560d38d9f600 0x560d38cb9180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x560d38d9ed00 msgr2=0x560d38cb8680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x560d38d9ed00 0x560d38cb8680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x560d38d9f180 msgr2=0x560d38cb8c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x560d38d9f180 0x560d38cb8c00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07cbe33640 1 -- 172.21.15.28:0/3795850831 reap_dead start 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x560d38d9fa80 msgr2=0x560d38cb8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x560d38d9fa80 0x560d38cb8100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x560d38d07170 tx=0x560d38f88840 comp rx=0 tx=0).stop 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9cd80 msgr2=0x560d38cb7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x560d39adbcb0 tx=0x560d38f16f60 comp rx=0 tx=0).stop 2024-04-25T19:13:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 shutdown_connections 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x560d38d9fa80 0x560d38cb8100 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/3795850831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9cd80 0x560d38cb7080 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 >> 172.21.15.28:0/3795850831 conn(0x560d38d9d200 msgr2=0x560d38ce8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 shutdown_connections 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/3795850831 wait complete. 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/610544592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9d680 msgr2=0x560d38cb6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9d680 0x560d38cb6000 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x560d39a21c80 tx=0x560d38d08120 comp rx=0 tx=0).stop 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/610544592 shutdown_connections 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x560d38d9df80 0x560d38cb6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x560d38d9db00 0x560d38cb6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 --2- 172.21.15.28:0/610544592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d38d9d680 0x560d38cb6000 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/610544592 >> 172.21.15.28:0/610544592 conn(0x560d38d9e400 msgr2=0x560d38ce9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/610544592 shutdown_connections 2024-04-25T19:13:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.879+0000 7f07ce810740 1 -- 172.21.15.28:0/610544592 wait complete. 2024-04-25T19:13:08.884 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'init', '--force-init', '--filesystem', 'recovery_fs', '--alternate-pool', 'recovery_fs_metadata'] in time 0:00:00.077536, result: 2024-04-25T19:13:08.884 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset session 2024-04-25T19:13:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.935+0000 7fd7cc242280 1 -- start start 2024-04-25T19:13:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.936+0000 7fd7cc242280 1 Processor -- start 2024-04-25T19:13:08.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1cad00 0x55fb4e1ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c8480 0x55fb4e1cd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c8900 0x55fb4e1cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cc242280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb4eef4000 con 0x55fb4e1c8900 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cc242280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fb4eef4180 con 0x55fb4e1c8480 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cc242280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55fb4eef4300 con 0x55fb4e1cad00 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c8900 0x55fb4e1cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c8900 0x55fb4e1cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51348/0 (socket says 172.21.15.28:51348) 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 -- 172.21.15.28:0/2254280728 learned_addr learned my addr 172.21.15.28:0/2254280728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c8480 0x55fb4e1cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7ca9f0640 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1cad00 0x55fb4e1ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 -- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1cad00 msgr2=0x55fb4e1ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1cad00 0x55fb4e1ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 -- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c8480 msgr2=0x55fb4e1cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c8480 0x55fb4e1cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 -- 172.21.15.28:0/2254280728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb4a11a780 con 0x55fb4e1c8900 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7ca9f0640 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1cad00 0x55fb4e1ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.937+0000 7fd7cb1f1640 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c8900 0x55fb4e1cc580 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x55fb4ef55470 tx=0x55fb4e2120c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f222a435b8219c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7c81eb640 1 -- 172.21.15.28:0/2254280728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fb4a10ad00 con 0x55fb4e1c8900 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7c81eb640 1 -- 172.21.15.28:0/2254280728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55fb4a110e00 con 0x55fb4e1c8900 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7c81eb640 1 -- 172.21.15.28:0/2254280728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fb4a10a680 con 0x55fb4e1c8900 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7cc242280 1 -- 172.21.15.28:0/2254280728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb4a11a3c0 con 0x55fb4e1c8900 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7cc242280 1 -- 172.21.15.28:0/2254280728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55fb4a11b860 con 0x55fb4e1c8900 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7cc242280 4 waiting for MDS map... 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7c81eb640 1 -- 172.21.15.28:0/2254280728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55fb4efeea00 con 0x55fb4e1c8900 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7c81eb640 1 -- 172.21.15.28:0/2254280728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x55fb4e16f180 con 0x55fb4e1c8900 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7cc242280 4 Got MDS map 25 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7cc242280 10 main: main 2024-04-25T19:13:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.938+0000 7fd7cc242280 4 main: connecting to RADOS... 2024-04-25T19:13:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.939+0000 7fd7cc242280 1 Processor -- start 2024-04-25T19:13:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.939+0000 7fd7cc242280 1 -- start start 2024-04-25T19:13:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1c9200 0x55fb4e1cc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9680 0x55fb4e1cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7cc242280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb4eef4480 con 0x55fb4e1c9b00 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7cc242280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fb4eef4600 con 0x55fb4e1c9680 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7cc242280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55fb4eef4780 con 0x55fb4e1c9200 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7ca1ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7ca1ef640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cdb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51352/0 (socket says 172.21.15.28:51352) 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.940+0000 7fd7ca1ef640 1 -- 172.21.15.28:0/1093620677 learned_addr learned my addr 172.21.15.28:0/1093620677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7ca1ef640 1 -- 172.21.15.28:0/1093620677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1c9200 msgr2=0x55fb4e1cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/1093620677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1c9200 0x55fb4e1cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7ca1ef640 1 -- 172.21.15.28:0/1093620677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9680 msgr2=0x55fb4e1cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/1093620677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9680 0x55fb4e1cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7ca1ef640 1 -- 172.21.15.28:0/1093620677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb4a11ba40 con 0x55fb4e1c9b00 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/1093620677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cdb80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x55fb4ef54de0 tx=0x55fb4e211f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6635cafe126e5882 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7c61e7640 1 -- 172.21.15.28:0/1093620677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fb4a10ab60 con 0x55fb4e1c9b00 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7c61e7640 1 -- 172.21.15.28:0/1093620677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55fb4a110a80 con 0x55fb4e1c9b00 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7c61e7640 1 -- 172.21.15.28:0/1093620677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fb4a10ad00 con 0x55fb4e1c9b00 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7cc242280 1 -- 172.21.15.28:0/1093620677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 msgr2=0x55fb4e1cdb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.941+0000 7fd7cc242280 1 --2- 172.21.15.28:0/1093620677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cdb80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x55fb4ef54de0 tx=0x55fb4e211f80 comp rx=0 tx=0).stop 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 -- 172.21.15.28:0/1093620677 shutdown_connections 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 --2- 172.21.15.28:0/1093620677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cdb80 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 --2- 172.21.15.28:0/1093620677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9680 0x55fb4e1cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 --2- 172.21.15.28:0/1093620677 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1c9200 0x55fb4e1cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 -- 172.21.15.28:0/1093620677 >> 172.21.15.28:0/1093620677 conn(0x55fb4e1c8d80 msgr2=0x55fb4e23e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 -- 172.21.15.28:0/1093620677 shutdown_connections 2024-04-25T19:13:08.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 -- 172.21.15.28:0/1093620677 wait complete. 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 Processor -- start 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.942+0000 7fd7cc242280 1 -- start start 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1c9680 0x55fb4e1cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9f80 0x55fb4e1cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55fb4eef5e00 con 0x55fb4e1c9b00 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55fb4eef4d80 con 0x55fb4e1c9f80 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55fb4eef4f00 con 0x55fb4e1c9680 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51354/0 (socket says 172.21.15.28:51354) 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 -- 172.21.15.28:0/913240734 learned_addr learned my addr 172.21.15.28:0/913240734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9f80 0x55fb4e1cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1c9680 0x55fb4e1cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 -- 172.21.15.28:0/913240734 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1c9680 msgr2=0x55fb4e1cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1c9680 0x55fb4e1cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 -- 172.21.15.28:0/913240734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9f80 msgr2=0x55fb4e1cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9f80 0x55fb4e1cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 -- 172.21.15.28:0/913240734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb4a11a780 con 0x55fb4e1c9b00 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c9f80 0x55fb4e1cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7ca9f0640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cd600 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x55fb4effba70 tx=0x55fb4e2127e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7c49e4640 1 -- 172.21.15.28:0/913240734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fb4a10ab60 con 0x55fb4e1c9b00 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7c49e4640 1 -- 172.21.15.28:0/913240734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55fb4a110a80 con 0x55fb4e1c9b00 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55fb4a11a3c0 con 0x55fb4e1c9b00 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7c49e4640 1 -- 172.21.15.28:0/913240734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55fb4a10ad00 con 0x55fb4e1c9b00 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb4a11bc20 con 0x55fb4e1c9b00 2024-04-25T19:13:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.944+0000 7fd7cc242280 4 main: resolving pool 7 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.946+0000 7fd7c49e4640 1 -- 172.21.15.28:0/913240734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x55fb4f043200 con 0x55fb4e1c9b00 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.946+0000 7fd7c49e4640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55fb4e2aba80 0x55fb4e1ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.946+0000 7fd7c49e4640 1 -- 172.21.15.28:0/913240734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55fb4f066a00 con 0x55fb4e1c9b00 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.946+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55fb4e2aba80 0x55fb4e1ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.946+0000 7fd7cc242280 4 main: creating IoCtx.. 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.946+0000 7fd7cc242280 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55fb4e1ca400 0x55fb4e1ce680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.946+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:1 7.b 7:d0630e4c:::mds0_sessionmap:head [omap-clear,omap-set-header in=14b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55fb4f064c00 con 0x55fb4e1ca400 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.946+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55fb4e1ca400 0x55fb4e1ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.947+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55fb4e2aba80 0x55fb4e1ce100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55fb4effa900 tx=0x55fb4e211f20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.947+0000 7fd7ca1ef640 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55fb4e1ca400 0x55fb4e1ce680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:08.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7ca1ef640 1 -- 172.21.15.28:0/913240734 <== osd.1 v2:172.21.15.28:6808/1898594658 1 ==== osd_op_reply(1 mds0_sessionmap [omap-clear,omap-set-header] v46'1 uv1 ondisk = 0) ==== 201+0+0 (crc 0 0 0) 0x55fb4e1fa000 con 0x55fb4e1ca400 2024-04-25T19:13:08.953 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "0": { 2024-04-25T19:13:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "data": {}, 2024-04-25T19:13:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: "result": 0 2024-04-25T19:13:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:13:08.953 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:08.953 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55fb4e1ca400 msgr2=0x55fb4e1ce680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55fb4e1ca400 0x55fb4e1ce680 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55fb4e2aba80 msgr2=0x55fb4e1ce100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55fb4e2aba80 0x55fb4e1ce100 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x55fb4effa900 tx=0x55fb4e211f20 comp rx=0 tx=0).stop 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 msgr2=0x55fb4e1cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 --2- 172.21.15.28:0/913240734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c9b00 0x55fb4e1cd600 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x55fb4effba70 tx=0x55fb4e2127e0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7ca9f0640 1 -- 172.21.15.28:0/913240734 reap_dead start 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 shutdown_connections 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 >> 172.21.15.28:0/913240734 conn(0x55fb4e1c9200 msgr2=0x55fb4e23e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 shutdown_connections 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/913240734 wait complete. 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/2254280728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c8900 msgr2=0x55fb4e1cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c8900 0x55fb4e1cc580 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x55fb4ef55470 tx=0x55fb4e2120c0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/2254280728 shutdown_connections 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb4e1c8900 0x55fb4e1cc580 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55fb4e1c8480 0x55fb4e1cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 --2- 172.21.15.28:0/2254280728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55fb4e1cad00 0x55fb4e1ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:08.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/2254280728 >> 172.21.15.28:0/2254280728 conn(0x55fb4e2ab600 msgr2=0x55fb4e23f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/2254280728 shutdown_connections 2024-04-25T19:13:08.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:08.949+0000 7fd7cc242280 1 -- 172.21.15.28:0/2254280728 wait complete. 2024-04-25T19:13:08.957 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'session'] in time 0:00:00.072396, result: { "0": { "data": {}, "result": 0 } } 2024-04-25T19:13:08.957 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset snap 2024-04-25T19:13:09.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.006+0000 7fadcf00e280 1 -- start start 2024-04-25T19:13:09.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.006+0000 7fadcf00e280 1 Processor -- start 2024-04-25T19:13:09.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d511da80 0x55d2d503e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d511d180 0x55d2d503e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a000 0x55d2d503eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcf00e280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d2d5d94180 con 0x55d2d5e4a000 2024-04-25T19:13:09.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcf00e280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d2d5d94300 con 0x55d2d511d180 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcf00e280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55d2d5d94480 con 0x55d2d511da80 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a000 0x55d2d503eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a000 0x55d2d503eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51380/0 (socket says 172.21.15.28:51380) 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 -- 172.21.15.28:0/1919742703 learned_addr learned my addr 172.21.15.28:0/1919742703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadccaaa640 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d511da80 0x55d2d503e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d511d180 0x55d2d503e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 -- 172.21.15.28:0/1919742703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d511da80 msgr2=0x55d2d503e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d511da80 0x55d2d503e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 -- 172.21.15.28:0/1919742703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d511d180 msgr2=0x55d2d503e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d511d180 0x55d2d503e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 -- 172.21.15.28:0/1919742703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2d0f8c780 con 0x55d2d5e4a000 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.008+0000 7fadcd2ab640 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a000 0x55d2d503eb00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55d2d5db93b0 tx=0x55d2d5084120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef895bb907a7f651 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadca2a5640 1 -- 172.21.15.28:0/1919742703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d2d0f7c680 con 0x55d2d5e4a000 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadca2a5640 1 -- 172.21.15.28:0/1919742703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d2d0f82fc0 con 0x55d2d5e4a000 2024-04-25T19:13:09.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadca2a5640 1 -- 172.21.15.28:0/1919742703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d2d0f7cb60 con 0x55d2d5e4a000 2024-04-25T19:13:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadcf00e280 1 -- 172.21.15.28:0/1919742703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2d0f8c3c0 con 0x55d2d5e4a000 2024-04-25T19:13:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadcf00e280 1 -- 172.21.15.28:0/1919742703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55d2d0f8d680 con 0x55d2d5e4a000 2024-04-25T19:13:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadcf00e280 4 waiting for MDS map... 2024-04-25T19:13:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadca2a5640 1 -- 172.21.15.28:0/1919742703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55d2d5e68600 con 0x55d2d5e4a000 2024-04-25T19:13:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadca2a5640 1 -- 172.21.15.28:0/1919742703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x55d2d4fe1180 con 0x55d2d5e4a000 2024-04-25T19:13:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadcf00e280 4 Got MDS map 25 2024-04-25T19:13:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadcf00e280 10 main: main 2024-04-25T19:13:09.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.009+0000 7fadcf00e280 4 main: connecting to RADOS... 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.010+0000 7fadcf00e280 1 Processor -- start 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.010+0000 7fadcf00e280 1 -- start start 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4a900 0x55d2d503f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 0x55d2d503f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4b200 0x55d2d503fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcf00e280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d2d5d94600 con 0x55d2d5e4b200 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcf00e280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d2d5d94780 con 0x55d2d5e4ad80 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcf00e280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55d2d5d94900 con 0x55d2d5e4a900 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcc2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4b200 0x55d2d503fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcc2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4b200 0x55d2d503fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51384/0 (socket says 172.21.15.28:51384) 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcc2a9640 1 -- 172.21.15.28:0/2406616932 learned_addr learned my addr 172.21.15.28:0/2406616932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.011+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4a900 0x55d2d503f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadccaaa640 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 0x55d2d503f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadcc2a9640 1 -- 172.21.15.28:0/2406616932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4a900 msgr2=0x55d2d503f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4a900 0x55d2d503f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadcc2a9640 1 -- 172.21.15.28:0/2406616932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 msgr2=0x55d2d503f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 0x55d2d503f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadcc2a9640 1 -- 172.21.15.28:0/2406616932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2d0f8d4a0 con 0x55d2d5e4b200 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4b200 0x55d2d503fb80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x55d2d5d648d0 tx=0x55d2d5d67860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff4d592a6116eee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadc82a1640 1 -- 172.21.15.28:0/2406616932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d2d0f7c680 con 0x55d2d5e4b200 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadc82a1640 1 -- 172.21.15.28:0/2406616932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d2d0f82a80 con 0x55d2d5e4b200 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadc82a1640 1 -- 172.21.15.28:0/2406616932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d2d0f7cb60 con 0x55d2d5e4b200 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadcf00e280 1 -- 172.21.15.28:0/2406616932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4b200 msgr2=0x55d2d503fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.012+0000 7fadcf00e280 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4b200 0x55d2d503fb80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x55d2d5d648d0 tx=0x55d2d5d67860 comp rx=0 tx=0).stop 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 -- 172.21.15.28:0/2406616932 shutdown_connections 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4b200 0x55d2d503fb80 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 0x55d2d503f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 --2- 172.21.15.28:0/2406616932 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4a900 0x55d2d503f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 -- 172.21.15.28:0/2406616932 >> 172.21.15.28:0/2406616932 conn(0x55d2d5e4a480 msgr2=0x55d2d50b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 -- 172.21.15.28:0/2406616932 shutdown_connections 2024-04-25T19:13:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 -- 172.21.15.28:0/2406616932 wait complete. 2024-04-25T19:13:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 Processor -- start 2024-04-25T19:13:09.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.013+0000 7fadcf00e280 1 -- start start 2024-04-25T19:13:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a900 0x55d2d503f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 0x55d2d503f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4b200 0x55d2d503fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d2d5d94900 con 0x55d2d5e4a900 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d2d5d94a80 con 0x55d2d5e4ad80 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55d2d5d94780 con 0x55d2d5e4b200 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a900 0x55d2d503f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a900 0x55d2d503f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51396/0 (socket says 172.21.15.28:51396) 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 -- 172.21.15.28:0/374492988 learned_addr learned my addr 172.21.15.28:0/374492988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadccaaa640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 0x55d2d503f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4b200 0x55d2d503fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 -- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4b200 msgr2=0x55d2d503fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d5e4b200 0x55d2d503fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 -- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 msgr2=0x55d2d503f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d5e4ad80 0x55d2d503f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 -- 172.21.15.28:0/374492988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2d0f8c780 con 0x55d2d5e4a900 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a900 0x55d2d503f080 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x55d2d5d65b90 tx=0x55d2d5d678c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadc6a9e640 1 -- 172.21.15.28:0/374492988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d2d0f7c4e0 con 0x55d2d5e4a900 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadc6a9e640 1 -- 172.21.15.28:0/374492988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d2d0f82a80 con 0x55d2d5e4a900 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d2d0f8c3c0 con 0x55d2d5e4a900 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadc6a9e640 1 -- 172.21.15.28:0/374492988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d2d0f7c680 con 0x55d2d5e4a900 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2d0f8c000 con 0x55d2d5e4a900 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.015+0000 7fadcf00e280 4 main: resolving pool 7 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.017+0000 7fadc6a9e640 1 -- 172.21.15.28:0/374492988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x55d2d5eb2900 con 0x55d2d5e4a900 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.017+0000 7fadc6a9e640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d2d5e4d600 0x55d2d5040100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.017+0000 7fadc6a9e640 1 -- 172.21.15.28:0/374492988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55d2d5eeac00 con 0x55d2d5e4a900 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.017+0000 7fadcf00e280 4 main: creating IoCtx.. 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.017+0000 7fadccaaa640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d2d5e4d600 0x55d2d5040100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.017+0000 7fadcf00e280 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55d2d5e4b680 0x55d2d5040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.017+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:1 7.d 7:b50e409b:::mds_snaptable:head [writefull 0~70 in=70b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55d2d5ee4c00 con 0x55d2d5e4b680 2024-04-25T19:13:09.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.018+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55d2d5e4b680 0x55d2d5040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.018+0000 7fadccaaa640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d2d5e4d600 0x55d2d5040100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x55d2d0f8f500 tx=0x55d2d50828a0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.018+0000 7fadcc2a9640 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55d2d5e4b680 0x55d2d5040680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcc2a9640 1 -- 172.21.15.28:0/374492988 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== osd_op_reply(1 mds_snaptable [writefull 0~70] v46'1 uv1 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x55d2d506c000 con 0x55d2d5e4b680 2024-04-25T19:13:09.023 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:09.023 INFO:teuthology.orchestra.run.smithi028.stdout: "result": 0 2024-04-25T19:13:09.023 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:09.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55d2d5e4b680 msgr2=0x55d2d5040680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55d2d5e4b680 0x55d2d5040680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d2d5e4d600 msgr2=0x55d2d5040100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d2d5e4d600 0x55d2d5040100 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x55d2d0f8f500 tx=0x55d2d50828a0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a900 msgr2=0x55d2d503f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 --2- 172.21.15.28:0/374492988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a900 0x55d2d503f080 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x55d2d5d65b90 tx=0x55d2d5d678c0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadccaaa640 1 -- 172.21.15.28:0/374492988 reap_dead start 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 shutdown_connections 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 >> 172.21.15.28:0/374492988 conn(0x55d2d5e4a480 msgr2=0x55d2d50b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 shutdown_connections 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 -- 172.21.15.28:0/374492988 wait complete. 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 -- 172.21.15.28:0/1919742703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a000 msgr2=0x55d2d503eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.020+0000 7fadcf00e280 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a000 0x55d2d503eb00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x55d2d5db93b0 tx=0x55d2d5084120 comp rx=0 tx=0).stop 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.021+0000 7fadcf00e280 1 -- 172.21.15.28:0/1919742703 shutdown_connections 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.021+0000 7fadcf00e280 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2d5e4a000 0x55d2d503eb00 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.021+0000 7fadcf00e280 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d2d511d180 0x55d2d503e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.021+0000 7fadcf00e280 1 --2- 172.21.15.28:0/1919742703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d2d511da80 0x55d2d503e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.021+0000 7fadcf00e280 1 -- 172.21.15.28:0/1919742703 >> 172.21.15.28:0/1919742703 conn(0x55d2d511d600 msgr2=0x55d2d50b3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.021+0000 7fadcf00e280 1 -- 172.21.15.28:0/1919742703 shutdown_connections 2024-04-25T19:13:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.021+0000 7fadcf00e280 1 -- 172.21.15.28:0/1919742703 wait complete. 2024-04-25T19:13:09.027 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'snap'] in time 0:00:00.069972, result: { "result": 0 } 2024-04-25T19:13:09.027 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 recovery_fs:0 reset inode 2024-04-25T19:13:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.078+0000 7f2660613280 1 -- start start 2024-04-25T19:13:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.078+0000 7f2660613280 1 Processor -- start 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f2660613280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069db3a80 0x55f06a9f6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f2660613280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069db3180 0x55f06a9f6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f2660613280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4000 0x55f06a9f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f2660613280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f06a9fe180 con 0x55f069cd4000 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f2660613280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f06a9fe300 con 0x55f069db3180 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f2660613280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55f06a9fe480 con 0x55f069db3a80 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f265ea1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4000 0x55f06a9f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f265ea1e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4000 0x55f06a9f6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51408/0 (socket says 172.21.15.28:51408) 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f265ea1e640 1 -- 172.21.15.28:0/3993971467 learned_addr learned my addr 172.21.15.28:0/3993971467 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f265da1c640 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069db3180 0x55f06a9f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.079+0000 7f265e21d640 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069db3a80 0x55f06a9f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ea1e640 1 -- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069db3a80 msgr2=0x55f06a9f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ea1e640 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069db3a80 0x55f06a9f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ea1e640 1 -- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069db3180 msgr2=0x55f06a9f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ea1e640 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069db3180 0x55f06a9f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ea1e640 1 -- 172.21.15.28:0/3993971467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f065c22780 con 0x55f069cd4000 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265da1c640 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069db3180 0x55f06a9f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ea1e640 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4000 0x55f06a9f6b00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x55f06aa553e0 tx=0x55f069d1a900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c350f632e3665aee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ba18640 1 -- 172.21.15.28:0/3993971467 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f065c12680 con 0x55f069cd4000 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ba18640 1 -- 172.21.15.28:0/3993971467 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55f065c18a80 con 0x55f069cd4000 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ba18640 1 -- 172.21.15.28:0/3993971467 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f065c12b60 con 0x55f069cd4000 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f2660613280 1 -- 172.21.15.28:0/3993971467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f065c223c0 con 0x55f069cd4000 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f2660613280 1 -- 172.21.15.28:0/3993971467 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55f065c23680 con 0x55f069cd4000 2024-04-25T19:13:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f2660613280 4 waiting for MDS map... 2024-04-25T19:13:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ba18640 1 -- 172.21.15.28:0/3993971467 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55f06aafca00 con 0x55f069cd4000 2024-04-25T19:13:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f265ba18640 1 -- 172.21.15.28:0/3993971467 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x55f069c77180 con 0x55f069cd4000 2024-04-25T19:13:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f2660613280 4 Got MDS map 25 2024-04-25T19:13:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.080+0000 7f2660613280 10 main: main 2024-04-25T19:13:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.081+0000 7f2660613280 4 main: connecting to RADOS... 2024-04-25T19:13:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.081+0000 7f2660613280 1 Processor -- start 2024-04-25T19:13:09.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.081+0000 7f2660613280 1 -- start start 2024-04-25T19:13:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.082+0000 7f2660613280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 0x55f06a9f7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f2660613280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4d80 0x55f06a9f7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f2660613280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd5200 0x55f06a9f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f2660613280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f06a9fe600 con 0x55f069cd4d80 2024-04-25T19:13:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f2660613280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f06a9fe780 con 0x55f069cd4900 2024-04-25T19:13:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f2660613280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55f06a9fe900 con 0x55f069cd5200 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4d80 0x55f06a9f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4d80 0x55f06a9f7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51416/0 (socket says 172.21.15.28:51416) 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 -- 172.21.15.28:0/247952632 learned_addr learned my addr 172.21.15.28:0/247952632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265da1c640 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd5200 0x55f06a9f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 -- 172.21.15.28:0/247952632 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd5200 msgr2=0x55f06a9f7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd5200 0x55f06a9f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265da1c640 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 0x55f06a9f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 -- 172.21.15.28:0/247952632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 msgr2=0x55f06a9f7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 0x55f06a9f7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 -- 172.21.15.28:0/247952632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f065c234a0 con 0x55f069cd4d80 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f265e21d640 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4d80 0x55f06a9f7600 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x55f06aaeea50 tx=0x55f06ab0b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36378bb4988c7a96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f2659a14640 1 -- 172.21.15.28:0/247952632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f065c12680 con 0x55f069cd4d80 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f2659a14640 1 -- 172.21.15.28:0/247952632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55f065c18fc0 con 0x55f069cd4d80 2024-04-25T19:13:09.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.083+0000 7f2659a14640 1 -- 172.21.15.28:0/247952632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f065c12b60 con 0x55f069cd4d80 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 -- 172.21.15.28:0/247952632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4d80 msgr2=0x55f06a9f7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4d80 0x55f06a9f7600 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x55f06aaeea50 tx=0x55f06ab0b800 comp rx=0 tx=0).stop 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 -- 172.21.15.28:0/247952632 shutdown_connections 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd5200 0x55f06a9f7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4d80 0x55f06a9f7600 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 --2- 172.21.15.28:0/247952632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 0x55f06a9f7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 -- 172.21.15.28:0/247952632 >> 172.21.15.28:0/247952632 conn(0x55f069cd4480 msgr2=0x55f069d49800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 -- 172.21.15.28:0/247952632 shutdown_connections 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.084+0000 7f2660613280 1 -- 172.21.15.28:0/247952632 wait complete. 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.085+0000 7f2660613280 1 Processor -- start 2024-04-25T19:13:09.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.085+0000 7f2660613280 1 -- start start 2024-04-25T19:13:09.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f2660613280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 0x55f06a9f7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f2660613280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd4d80 0x55f06a9f7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f2660613280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd5200 0x55f06a9f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f2660613280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f06a9fe900 con 0x55f069cd5200 2024-04-25T19:13:09.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f2660613280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f06a9fea80 con 0x55f069cd4900 2024-04-25T19:13:09.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f2660613280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55f06a9fe780 con 0x55f069cd4d80 2024-04-25T19:13:09.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd5200 0x55f06a9f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd5200 0x55f06a9f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51420/0 (socket says 172.21.15.28:51420) 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 -- 172.21.15.28:0/4037818719 learned_addr learned my addr 172.21.15.28:0/4037818719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265e21d640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd4d80 0x55f06a9f7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 0x55f06a9f7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 -- 172.21.15.28:0/4037818719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd4d80 msgr2=0x55f06a9f7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069cd4d80 0x55f06a9f7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 -- 172.21.15.28:0/4037818719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 msgr2=0x55f06a9f7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069cd4900 0x55f06a9f7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 -- 172.21.15.28:0/4037818719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f065c22780 con 0x55f069cd5200 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.086+0000 7f265da1c640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd5200 0x55f06a9f7b80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x55f06aaefce0 tx=0x55f069d18660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.087+0000 7f2658211640 1 -- 172.21.15.28:0/4037818719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f065c129c0 con 0x55f069cd5200 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.087+0000 7f2658211640 1 -- 172.21.15.28:0/4037818719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55f065c18e00 con 0x55f069cd5200 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.087+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f065c223c0 con 0x55f069cd5200 2024-04-25T19:13:09.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.087+0000 7f2658211640 1 -- 172.21.15.28:0/4037818719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f065c12680 con 0x55f069cd5200 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.087+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f065c23680 con 0x55f069cd5200 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.087+0000 7f2660613280 4 main: resolving pool 7 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.088+0000 7f2658211640 1 -- 172.21.15.28:0/4037818719 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x55f06ab46900 con 0x55f069cd5200 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f2658211640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f069cd7600 0x55f06a9f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f2658211640 1 -- 172.21.15.28:0/4037818719 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55f06ab62c00 con 0x55f069cd5200 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f265e21d640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f069cd7600 0x55f06a9f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f2660613280 4 main: creating IoCtx.. 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f2660613280 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f069cd5680 0x55f06a9f8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:1 7.13 7:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f06ab61c00 con 0x55f069cd5680 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f265da1c640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f069cd5680 0x55f06a9f8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f265e21d640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f069cd7600 0x55f06a9f8100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55f06aaeea50 tx=0x55f069d18780 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.089+0000 7f265da1c640 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f069cd5680 0x55f06a9f8680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.091+0000 7f265da1c640 1 -- 172.21.15.28:0/4037818719 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v46'2 uv2 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x55f069d02000 con 0x55f069cd5680 2024-04-25T19:13:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:09.095 INFO:teuthology.orchestra.run.smithi028.stdout: "0": { 2024-04-25T19:13:09.095 INFO:teuthology.orchestra.run.smithi028.stdout: "data": {}, 2024-04-25T19:13:09.095 INFO:teuthology.orchestra.run.smithi028.stdout: "result": 0 2024-04-25T19:13:09.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-25T19:13:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:09.095 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:09.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.091+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f069cd5680 msgr2=0x55f06a9f8680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.091+0000 7f2660613280 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f069cd5680 0x55f06a9f8680 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.091+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f069cd7600 msgr2=0x55f06a9f8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.091+0000 7f2660613280 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f069cd7600 0x55f06a9f8100 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55f06aaeea50 tx=0x55f069d18780 comp rx=0 tx=0).stop 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.091+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd5200 msgr2=0x55f06a9f7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.091+0000 7f2660613280 1 --2- 172.21.15.28:0/4037818719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd5200 0x55f06a9f7b80 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x55f06aaefce0 tx=0x55f069d18660 comp rx=0 tx=0).stop 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f265e21d640 1 -- 172.21.15.28:0/4037818719 reap_dead start 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 shutdown_connections 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 >> 172.21.15.28:0/4037818719 conn(0x55f069cd4480 msgr2=0x55f069d49800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 shutdown_connections 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/4037818719 wait complete. 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/3993971467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4000 msgr2=0x55f06a9f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4000 0x55f06a9f6b00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x55f06aa553e0 tx=0x55f069d1a900 comp rx=0 tx=0).stop 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/3993971467 shutdown_connections 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f069cd4000 0x55f06a9f6b00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f069db3180 0x55f06a9f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 --2- 172.21.15.28:0/3993971467 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f069db3a80 0x55f06a9f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/3993971467 >> 172.21.15.28:0/3993971467 conn(0x55f069db3600 msgr2=0x55f069d49000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/3993971467 shutdown_connections 2024-04-25T19:13:09.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.092+0000 7f2660613280 1 -- 172.21.15.28:0/3993971467 wait complete. 2024-04-25T19:13:09.099 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'recovery_fs:0', 'reset', 'inode'] in time 0:00:00.071808, result: { "0": { "data": {}, "result": 0 } } 2024-04-25T19:13:09.099 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents --alternate-pool recovery_fs_metadata --filesystem cephfs cephfs_data 2024-04-25T19:13:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.152+0000 7f663af20740 1 -- start start 2024-04-25T19:13:09.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.152+0000 7f663af20740 1 Processor -- start 2024-04-25T19:13:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.153+0000 7f663af20740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222d680 0x557d22e78580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.153+0000 7f663af20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222db00 0x557d22e78000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.153+0000 7f663af20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222df80 0x557d22e78b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.153+0000 7f663af20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d22152180 con 0x557d2222df80 2024-04-25T19:13:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.153+0000 7f663af20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557d22152300 con 0x557d2222db00 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.153+0000 7f663af20740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x557d22152480 con 0x557d2222d680 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222df80 0x557d22e78b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222df80 0x557d22e78b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51432/0 (socket says 172.21.15.28:51432) 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 -- 172.21.15.28:0/1177470161 learned_addr learned my addr 172.21.15.28:0/1177470161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66386ac640 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222db00 0x557d22e78000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f6638ead640 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222d680 0x557d22e78580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 -- 172.21.15.28:0/1177470161 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222d680 msgr2=0x557d22e78580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222d680 0x557d22e78580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 -- 172.21.15.28:0/1177470161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222db00 msgr2=0x557d22e78000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222db00 0x557d22e78000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 -- 172.21.15.28:0/1177470161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d1e091680 con 0x557d2222df80 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66396ae640 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222df80 0x557d22e78b00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x557d22ec3cb0 tx=0x557d22198a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceff01705377a92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66366a8640 1 -- 172.21.15.28:0/1177470161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d1e08a680 con 0x557d2222df80 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66366a8640 1 -- 172.21.15.28:0/1177470161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557d220dc1c0 con 0x557d2222df80 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f66366a8640 1 -- 172.21.15.28:0/1177470161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d1e08ab60 con 0x557d2222df80 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f663af20740 1 -- 172.21.15.28:0/1177470161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d1e091860 con 0x557d2222df80 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f663af20740 1 -- 172.21.15.28:0/1177470161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x557d1e0912c0 con 0x557d2222df80 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.154+0000 7f663af20740 4 waiting for MDS map... 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.155+0000 7f66366a8640 1 -- 172.21.15.28:0/1177470161 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x557d22f84600 con 0x557d2222df80 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.155+0000 7f66366a8640 1 -- 172.21.15.28:0/1177470161 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x557d220ed180 con 0x557d2222df80 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.155+0000 7f663af20740 4 Got MDS map 25 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.155+0000 7f663af20740 4 datascan.main: Using metadata pool output 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.155+0000 7f663af20740 4 datascan.main: connecting to RADOS... 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.156+0000 7f663af20740 1 Processor -- start 2024-04-25T19:13:09.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.156+0000 7f663af20740 1 -- start start 2024-04-25T19:13:09.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f663af20740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222cd80 0x557d22e79080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f663af20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 0x557d22e79600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f663af20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222e880 0x557d22e79b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f663af20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d22152600 con 0x557d2222e880 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f663af20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557d22152780 con 0x557d2222c900 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f663af20740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x557d22152900 con 0x557d2222cd80 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222e880 0x557d22e79b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222e880 0x557d22e79b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51442/0 (socket says 172.21.15.28:51442) 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 -- 172.21.15.28:0/141270300 learned_addr learned my addr 172.21.15.28:0/141270300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 -- 172.21.15.28:0/141270300 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222cd80 msgr2=0x557d22e79080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 --2- 172.21.15.28:0/141270300 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222cd80 0x557d22e79080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f6638ead640 1 --2- 172.21.15.28:0/141270300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 0x557d22e79600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 -- 172.21.15.28:0/141270300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 msgr2=0x557d22e79600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 --2- 172.21.15.28:0/141270300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 0x557d22e79600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 -- 172.21.15.28:0/141270300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d1e0914a0 con 0x557d2222e880 2024-04-25T19:13:09.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.157+0000 7f66386ac640 1 --2- 172.21.15.28:0/141270300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222e880 0x557d22e79b80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x557d22f87260 tx=0x557d22194c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=452c15400d9f320d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f66346a4640 1 -- 172.21.15.28:0/141270300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d1e08a9c0 con 0x557d2222e880 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f66346a4640 1 -- 172.21.15.28:0/141270300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557d220dce00 con 0x557d2222e880 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f66346a4640 1 -- 172.21.15.28:0/141270300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d1e08a680 con 0x557d2222e880 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 -- 172.21.15.28:0/141270300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222e880 msgr2=0x557d22e79b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 --2- 172.21.15.28:0/141270300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222e880 0x557d22e79b80 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x557d22f87260 tx=0x557d22194c60 comp rx=0 tx=0).stop 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 -- 172.21.15.28:0/141270300 shutdown_connections 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 --2- 172.21.15.28:0/141270300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222e880 0x557d22e79b80 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 --2- 172.21.15.28:0/141270300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 0x557d22e79600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 --2- 172.21.15.28:0/141270300 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222cd80 0x557d22e79080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 -- 172.21.15.28:0/141270300 >> 172.21.15.28:0/141270300 conn(0x557d2222d200 msgr2=0x557d2217b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 -- 172.21.15.28:0/141270300 shutdown_connections 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.158+0000 7f663af20740 1 -- 172.21.15.28:0/141270300 wait complete. 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.159+0000 7f663af20740 1 Processor -- start 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.159+0000 7f663af20740 1 -- start start 2024-04-25T19:13:09.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f663af20740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 0x557d22e79600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f663af20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222cd80 0x557d22e79080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f663af20740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222e880 0x557d22e79b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f663af20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d220f4180 con 0x557d2222cd80 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f663af20740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x557d22153e00 con 0x557d2222c900 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f663af20740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x557d22152d80 con 0x557d2222e880 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f6638ead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222cd80 0x557d22e79080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f6638ead640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222cd80 0x557d22e79080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51458/0 (socket says 172.21.15.28:51458) 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.160+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 learned_addr learned my addr 172.21.15.28:0/1389534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 0x557d22e79600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222e880 0x557d22e79b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222e880 msgr2=0x557d22e79b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6638ead640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222e880 0x557d22e79b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 msgr2=0x557d22e79600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6638ead640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222c900 0x557d22e79600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d1e091680 con 0x557d2222cd80 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222e880 0x557d22e79b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6638ead640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222cd80 0x557d22e79080 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x557d22f87f80 tx=0x557d22f91800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6632ea1640 1 -- 172.21.15.28:0/1389534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d1e08a9c0 con 0x557d2222cd80 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6632ea1640 1 -- 172.21.15.28:0/1389534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x557d220dcc40 con 0x557d2222cd80 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557d1e091860 con 0x557d2222cd80 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f6632ea1640 1 -- 172.21.15.28:0/1389534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557d1e08a680 con 0x557d2222cd80 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d1e0910e0 con 0x557d2222cd80 2024-04-25T19:13:09.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.161+0000 7f663af20740 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-04-25T19:13:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f6632ea1640 1 -- 172.21.15.28:0/1389534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x557d22fc6900 con 0x557d2222cd80 2024-04-25T19:13:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f6632ea1640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x557d2222fa80 0x557d22e7a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f6632ea1640 1 -- 172.21.15.28:0/1389534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x557d22feaa00 con 0x557d2222cd80 2024-04-25T19:13:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x557d2222fa80 0x557d22e7a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f663af20740 4 datascan.main: data pool 'cephfs_data' has ID 6 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f663af20740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x557d2222ed00 0x557d22e7a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:1 6.0 6:00000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe8c00 con 0x557d2222ed00 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.163+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x557d2222ed00 0x557d22e7a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.164+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x557d2222fa80 0x557d22e7a100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x557d22f87470 tx=0x557d22195f20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.164+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x557d2222ed00 0x557d22e7a680 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.164+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== osd_op_reply(1 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d2222ed00 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.164+0000 7f6633ea3640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x557d22494000 0x557d2249c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.164+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:2 6.20 6:04000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9400 con 0x557d22494000 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.164+0000 7f66396ae640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x557d22494000 0x557d2249c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.165+0000 7f66396ae640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x557d22494000 0x557d2249c000 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.165+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.1 v2:172.21.15.28:6808/1898594658 1 ==== osd_op_reply(2 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494000 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.165+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:3 6.10 6:08000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9800 con 0x557d2222ed00 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.165+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.0 v2:172.21.15.28:6800/1137456103 2 ==== osd_op_reply(3 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d2222ed00 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.165+0000 7f6633ea3640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x557d22494480 0x557d2249c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.165+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:4 6.30 6:0c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9c00 con 0x557d22494480 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.166+0000 7f6638ead640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x557d22494480 0x557d2249c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.166+0000 7f6638ead640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x557d22494480 0x557d2249c580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.167+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 1 ==== osd_op_reply(4 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.167+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:5 6.8 6:10000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4000 con 0x557d22494480 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.167+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 2 ==== osd_op_reply(5 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.167+0000 7f6633ea3640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x557d22494900 0x557d2249cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.167+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:6 6.28 6:14000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4800 con 0x557d22494900 2024-04-25T19:13:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.167+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x557d22494900 0x557d2249cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.167+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x557d22494900 0x557d2249cb00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.168+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.2 v2:172.21.15.28:6816/3410596961 1 ==== osd_op_reply(6 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22494900 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.168+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:7 6.18 6:18000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4400 con 0x557d2222ed00 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.168+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.0 v2:172.21.15.28:6800/1137456103 3 ==== osd_op_reply(7 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d2222ed00 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.168+0000 7f6633ea3640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x557d22494d80 0x557d2249d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.168+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:8 6.38 6:1c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4c00 con 0x557d22494d80 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.168+0000 7f66396ae640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x557d22494d80 0x557d2249d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.168+0000 7f66396ae640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x557d22494d80 0x557d2249d080 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.169+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== osd_op_reply(8 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.169+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:9 6.4 6:20000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5400 con 0x557d22494000 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.169+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.1 v2:172.21.15.28:6808/1898594658 2 ==== osd_op_reply(9 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494000 2024-04-25T19:13:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.169+0000 7f6633ea3640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x557d22495200 0x557d2249d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.169+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:10 6.24 6:24000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5000 con 0x557d22495200 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.169+0000 7f6638ead640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x557d22495200 0x557d2249d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.170+0000 7f6638ead640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x557d22495200 0x557d2249d600 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.170+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 1 ==== osd_op_reply(10 [pgnls start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.170+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:11 6.14 6:28000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5800 con 0x557d22494900 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.170+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.2 v2:172.21.15.28:6816/3410596961 2 ==== osd_op_reply(11 [pgnls start_epoch 46 out=49b] v38'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22494900 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.170+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:12 6.34 6:2c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4800 con 0x557d2222ed00 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.171+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.0 v2:172.21.15.28:6800/1137456103 4 ==== osd_op_reply(12 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d2222ed00 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.171+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:13 6.c 6:30000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5c00 con 0x557d22494d80 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.171+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 2 ==== osd_op_reply(13 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.171+0000 7f6633ea3640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x557d22495680 0x557d2249db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.171+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:14 6.2c 6:34000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4400 con 0x557d22495680 2024-04-25T19:13:09.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.171+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x557d22495680 0x557d2249db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.172+0000 7f66386ac640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x557d22495680 0x557d2249db80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.172+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== osd_op_reply(14 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22495680 2024-04-25T19:13:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.172+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:15 6.1c 6:38000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4c00 con 0x557d22494480 2024-04-25T19:13:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.172+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 3 ==== osd_op_reply(15 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.172+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:16 6.3c 6:3c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe8c00 con 0x557d22494900 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.173+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.2 v2:172.21.15.28:6816/3410596961 3 ==== osd_op_reply(16 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22494900 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.173+0000 7f66346a4640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x557d22495b00 0x557d2249e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.173+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:17 6.2 6:40000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5400 con 0x557d22495b00 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.173+0000 7f66396ae640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x557d22495b00 0x557d2249e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.174+0000 7f66396ae640 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x557d22495b00 0x557d2249e100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.174+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.4 v2:172.21.15.53:6824/2055146181 1 ==== osd_op_reply(17 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22495b00 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.174+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:18 6.22 6:44000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9800 con 0x557d22495680 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.174+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.6 v2:172.21.15.53:6809/3653668189 2 ==== osd_op_reply(18 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22495680 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.174+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:19 6.12 6:48000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9400 con 0x557d22494480 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.175+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 4 ==== osd_op_reply(19 [pgnls start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.175+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:20 6.32 6:4c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4000 con 0x557d22495b00 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.175+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.4 v2:172.21.15.53:6824/2055146181 2 ==== osd_op_reply(20 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22495b00 2024-04-25T19:13:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.175+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:21 6.a 6:50000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22f83400 con 0x557d22495200 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.176+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 2 ==== osd_op_reply(21 [pgnls start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.176+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:22 6.2a 6:54000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5000 con 0x557d22495b00 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.176+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.4 v2:172.21.15.53:6824/2055146181 3 ==== osd_op_reply(22 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22495b00 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.176+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:23 6.1a 6:58000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5800 con 0x557d22495b00 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.177+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.4 v2:172.21.15.53:6824/2055146181 4 ==== osd_op_reply(23 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22495b00 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.177+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:24 6.3a 6:5c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4c00 con 0x557d22494000 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.177+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.1 v2:172.21.15.28:6808/1898594658 3 ==== osd_op_reply(24 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494000 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.177+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:25 6.6 6:60000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0000 con 0x557d22494d80 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.177+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 3 ==== osd_op_reply(25 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.177+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:26 6.26 6:64000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5c00 con 0x557d22494000 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.178+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.1 v2:172.21.15.28:6808/1898594658 4 ==== osd_op_reply(26 [pgnls start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494000 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.178+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:27 6.16 6:68000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0400 con 0x557d2222ed00 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.178+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.0 v2:172.21.15.28:6800/1137456103 5 ==== osd_op_reply(27 [pgnls start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d2222ed00 2024-04-25T19:13:09.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.178+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:28 6.36 6:6c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9c00 con 0x557d22495200 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.178+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 3 ==== osd_op_reply(28 [pgnls start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.178+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:29 6.e 6:70000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0800 con 0x557d22495b00 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.179+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.4 v2:172.21.15.53:6824/2055146181 5 ==== osd_op_reply(29 [pgnls start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22495b00 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.179+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:30 6.2e 6:74000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0000 con 0x557d22495680 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.179+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.6 v2:172.21.15.53:6809/3653668189 3 ==== osd_op_reply(30 [pgnls start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22495680 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.179+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:31 6.1e 6:78000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0c00 con 0x557d22495b00 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.180+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.4 v2:172.21.15.53:6824/2055146181 6 ==== osd_op_reply(31 [pgnls start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22495b00 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.180+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:32 6.3e 6:7c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5400 con 0x557d22494d80 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.180+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 4 ==== osd_op_reply(32 [pgnls start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.180+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:33 6.1 6:80000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d1000 con 0x557d22494000 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.180+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.1 v2:172.21.15.28:6808/1898594658 5 ==== osd_op_reply(33 [pgnls start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494000 2024-04-25T19:13:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.180+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:34 6.21 6:84000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4000 con 0x557d22494480 2024-04-25T19:13:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.181+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 5 ==== osd_op_reply(34 [pgnls start_epoch 46 out=49b] v38'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.181+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:35 6.11 6:88000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d1400 con 0x557d22494d80 2024-04-25T19:13:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.181+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 5 ==== osd_op_reply(35 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.181+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:36 6.31 6:8c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5000 con 0x557d22495200 2024-04-25T19:13:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.181+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 4 ==== osd_op_reply(36 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.181+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:37 6.9 6:90000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9400 con 0x557d2222ed00 2024-04-25T19:13:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.182+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.0 v2:172.21.15.28:6800/1137456103 6 ==== osd_op_reply(37 [pgnls start_epoch 46 out=49b] v38'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d2222ed00 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.182+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:38 6.29 6:94000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5800 con 0x557d22494480 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.182+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 6 ==== osd_op_reply(38 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.182+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:39 6.19 6:98000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9c00 con 0x557d22495200 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.182+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 5 ==== osd_op_reply(39 [pgnls start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.182+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:40 6.39 6:9c000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4c00 con 0x557d22495200 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.183+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 6 ==== osd_op_reply(40 [pgnls start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.183+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:41 6.5 6:a0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4000 con 0x557d22494480 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.183+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 7 ==== osd_op_reply(41 [pgnls start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.183+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:42 6.25 6:a4000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0400 con 0x557d22495680 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.183+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.6 v2:172.21.15.53:6809/3653668189 4 ==== osd_op_reply(42 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22495680 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.183+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:43 6.15 6:a8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5000 con 0x557d22494480 2024-04-25T19:13:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.184+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 8 ==== osd_op_reply(43 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.184+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:44 6.35 6:ac000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0000 con 0x557d22495200 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.184+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 7 ==== osd_op_reply(44 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.184+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:45 6.d 6:b0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4800 con 0x557d22495200 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.185+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 8 ==== osd_op_reply(45 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.185+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:46 6.2d 6:b4000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d1800 con 0x557d22494900 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.185+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.2 v2:172.21.15.28:6816/3410596961 4 ==== osd_op_reply(46 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22494900 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.185+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:47 6.1d 6:b8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4400 con 0x557d22494000 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.185+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.1 v2:172.21.15.28:6808/1898594658 6 ==== osd_op_reply(47 [pgnls start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494000 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.185+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:48 6.3d 6:bc000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d1c00 con 0x557d2222ed00 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.186+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.0 v2:172.21.15.28:6800/1137456103 7 ==== osd_op_reply(48 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d2222ed00 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.186+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:49 6.3 6:c0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22f83400 con 0x557d22494480 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.186+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 9 ==== osd_op_reply(49 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.186+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:50 6.23 6:c4000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe8c00 con 0x557d22495680 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.187+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.6 v2:172.21.15.53:6809/3653668189 5 ==== osd_op_reply(50 [pgnls start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22495680 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.187+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:51 6.13 6:c8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5c00 con 0x557d22494d80 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.187+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 6 ==== osd_op_reply(51 [pgnls start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.187+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:52 6.33 6:cc000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9800 con 0x557d22495200 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.187+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 9 ==== osd_op_reply(52 [pgnls start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.187+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:53 6.b 6:d0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5400 con 0x557d22494d80 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.188+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 7 ==== osd_op_reply(53 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.188+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:54 6.2b 6:d4000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0800 con 0x557d22495200 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.188+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 10 ==== osd_op_reply(54 [pgnls start_epoch 46 out=49b] v38'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.188+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:55 6.1b 6:d8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4400 con 0x557d22494d80 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.188+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 8 ==== osd_op_reply(55 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.188+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:56 6.3b 6:dc000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0c00 con 0x557d22494900 2024-04-25T19:13:09.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.189+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.2 v2:172.21.15.28:6816/3410596961 5 ==== osd_op_reply(56 [pgnls start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22494900 2024-04-25T19:13:09.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.189+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:57 6.7 6:e0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b5800 con 0x557d22495200 2024-04-25T19:13:09.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.189+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 11 ==== osd_op_reply(57 [pgnls start_epoch 46 out=81b] v38'8 uv8 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.189+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:58 6.7 6:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x557d22fe9000 con 0x557d22495200 2024-04-25T19:13:09.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.190+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 12 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv8 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.190+0000 7f663af20740 10 datascan.operator(): handling object 1099511627777.0 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.190+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:59 6.7 6:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x557d224d6000 con 0x557d22495200 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.192+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 13 ==== osd_op_reply(59 10000000001.00000000 [call] v46'9 uv9 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.192+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:60 6.27 6:e4000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d6400 con 0x557d22495680 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.192+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.6 v2:172.21.15.53:6809/3653668189 6 ==== osd_op_reply(60 [pgnls start_epoch 46 out=49b] v38'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22495680 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.192+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:61 6.17 6:e8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4c00 con 0x557d22495b00 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.192+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.4 v2:172.21.15.53:6824/2055146181 7 ==== osd_op_reply(61 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22495b00 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.192+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:62 6.37 6:ec000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d1000 con 0x557d2222ed00 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.193+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.0 v2:172.21.15.28:6800/1137456103 8 ==== osd_op_reply(62 [pgnls start_epoch 46 out=49b] v38'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d2222ed00 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.193+0000 7f66346a4640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:63 6.f 6:f0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224b4000 con 0x557d22494900 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.193+0000 7f66386ac640 1 -- 172.21.15.28:0/1389534 <== osd.2 v2:172.21.15.28:6816/3410596961 6 ==== osd_op_reply(63 [pgnls start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c000 con 0x557d22494900 2024-04-25T19:13:09.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.193+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:64 6.2f 6:f4000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d1400 con 0x557d22494480 2024-04-25T19:13:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.194+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 10 ==== osd_op_reply(64 [pgnls start_epoch 46 out=81b] v38'9 uv1 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.194+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:65 6.2f 6:f526278d:::10000000001.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x557d224d6800 con 0x557d22494480 2024-04-25T19:13:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.194+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.7 v2:172.21.15.53:6808/79451145 11 ==== osd_op_reply(65 10000000001.00000001 [stat out=16b] v0'0 uv1 ondisk = 0) ==== 164+0+16 (crc 0 0 0) 0x557d2217c780 con 0x557d22494480 2024-04-25T19:13:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.194+0000 7f663af20740 10 datascan.operator(): handling object 1099511627777.1 2024-04-25T19:13:09.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.194+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:66 6.7 6:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0400 con 0x557d22495200 2024-04-25T19:13:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 <== osd.5 v2:172.21.15.53:6800/2342659583 14 ==== osd_op_reply(66 10000000001.00000000 [call] v46'10 uv10 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x557d2217c780 con 0x557d22495200 2024-04-25T19:13:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:67 6.1f 6:f8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d1800 con 0x557d22494d80 2024-04-25T19:13:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 9 ==== osd_op_reply(67 [pgnls start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f6633ea3640 1 -- 172.21.15.28:0/1389534 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:68 6.3f 6:fc000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x557d224d0000 con 0x557d22494d80 2024-04-25T19:13:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f66396ae640 1 -- 172.21.15.28:0/1389534 <== osd.3 v2:172.21.15.28:6818/1928233777 10 ==== osd_op_reply(68 [pgnls start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x557d2217c500 con 0x557d22494d80 2024-04-25T19:13:09.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x557d2222ed00 msgr2=0x557d22e7a680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x557d2222ed00 0x557d22e7a680 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x557d22494000 msgr2=0x557d2249c000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x557d22494000 0x557d2249c000 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x557d22494900 msgr2=0x557d2249cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x557d22494900 0x557d2249cb00 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x557d22494d80 msgr2=0x557d2249d080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x557d22494d80 0x557d2249d080 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 reap_dead start 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x557d22495b00 msgr2=0x557d2249e100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x557d22495b00 0x557d2249e100 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x557d22495200 msgr2=0x557d2249d600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x557d22495200 0x557d2249d600 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x557d22495680 msgr2=0x557d2249db80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x557d22495680 0x557d2249db80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x557d22494480 msgr2=0x557d2249c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x557d22494480 0x557d2249c580 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f6638ead640 1 -- 172.21.15.28:0/1389534 reap_dead start 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x557d2222fa80 msgr2=0x557d22e7a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.196+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x557d2222fa80 0x557d22e7a100 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x557d22f87470 tx=0x557d22195f20 comp rx=0 tx=0).stop 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222cd80 msgr2=0x557d22e79080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222cd80 0x557d22e79080 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x557d22f87f80 tx=0x557d22f91800 comp rx=0 tx=0).stop 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 shutdown_connections 2024-04-25T19:13:09.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x557d2222fa80 0x557d22e7a100 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 --2- 172.21.15.28:0/1389534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222cd80 0x557d22e79080 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 >> 172.21.15.28:0/1389534 conn(0x557d2222d200 msgr2=0x557d2217b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 shutdown_connections 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1389534 wait complete. 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1177470161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222df80 msgr2=0x557d22e78b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222df80 0x557d22e78b00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x557d22ec3cb0 tx=0x557d22198a20 comp rx=0 tx=0).stop 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1177470161 shutdown_connections 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d2222df80 0x557d22e78b00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x557d2222db00 0x557d22e78000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 --2- 172.21.15.28:0/1177470161 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x557d2222d680 0x557d22e78580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1177470161 >> 172.21.15.28:0/1177470161 conn(0x557d2222e400 msgr2=0x557d2217b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1177470161 shutdown_connections 2024-04-25T19:13:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.197+0000 7f663af20740 1 -- 172.21.15.28:0/1177470161 wait complete. 2024-04-25T19:13:09.205 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', '--alternate-pool', 'recovery_fs_metadata', '--filesystem', 'cephfs', 'cephfs_data'] in time 0:00:00.106395, result: 2024-04-25T19:13:09.206 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --alternate-pool recovery_fs_metadata --filesystem cephfs --force-corrupt --force-init cephfs_data 2024-04-25T19:13:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.264+0000 7f8883210740 1 -- start start 2024-04-25T19:13:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.264+0000 7f8883210740 1 Processor -- start 2024-04-25T19:13:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.265+0000 7f8883210740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc1680 0x55753db9a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.265+0000 7f8883210740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc1b00 0x55753db9a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.265+0000 7f8883210740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc1f80 0x55753db9ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.265+0000 7f8883210740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55753dae6600 con 0x55753dbc1680 2024-04-25T19:13:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.265+0000 7f8883210740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55753dae6480 con 0x55753dbc1f80 2024-04-25T19:13:09.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.265+0000 7f8883210740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55753dae6300 con 0x55753dbc1b00 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.265+0000 7f8880fb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc1680 0x55753db9a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8880fb6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc1680 0x55753db9a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51484/0 (socket says 172.21.15.28:51484) 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8880fb6640 1 -- 172.21.15.28:0/2931074542 learned_addr learned my addr 172.21.15.28:0/2931074542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f88817b7640 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc1f80 0x55753db9ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f88807b5640 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc1b00 0x55753db9a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8880fb6640 1 -- 172.21.15.28:0/2931074542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc1b00 msgr2=0x55753db9a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8880fb6640 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc1b00 0x55753db9a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8880fb6640 1 -- 172.21.15.28:0/2931074542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc1f80 msgr2=0x55753db9ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8880fb6640 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc1f80 0x55753db9ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8880fb6640 1 -- 172.21.15.28:0/2931074542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557539a25680 con 0x55753dbc1680 2024-04-25T19:13:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8880fb6640 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc1680 0x55753db9a000 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x55753e845cb0 tx=0x55753db2c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7b0616cfb14c5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f887e7b1640 1 -- 172.21.15.28:0/2931074542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557539a1e680 con 0x55753dbc1680 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f887e7b1640 1 -- 172.21.15.28:0/2931074542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55753da70c40 con 0x55753dbc1680 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f887e7b1640 1 -- 172.21.15.28:0/2931074542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557539a1eb60 con 0x55753dbc1680 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8883210740 1 -- 172.21.15.28:0/2931074542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557539a254a0 con 0x55753dbc1680 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8883210740 1 -- 172.21.15.28:0/2931074542 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x557539a25a40 con 0x55753dbc1680 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f8883210740 4 waiting for MDS map... 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.266+0000 7f887e7b1640 1 -- 172.21.15.28:0/2931074542 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55753e90c600 con 0x55753dbc1680 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.267+0000 7f887e7b1640 1 -- 172.21.15.28:0/2931074542 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x55753da81180 con 0x55753dbc1680 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.267+0000 7f8883210740 4 Got MDS map 25 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.267+0000 7f8883210740 4 datascan.main: Using metadata pool output 2024-04-25T19:13:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.267+0000 7f8883210740 4 datascan.main: connecting to RADOS... 2024-04-25T19:13:09.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.268+0000 7f8883210740 1 Processor -- start 2024-04-25T19:13:09.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.268+0000 7f8883210740 1 -- start start 2024-04-25T19:13:09.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f8883210740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc0d80 0x55753db9b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f8883210740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0900 0x55753db9b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f8883210740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2880 0x55753db9bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f8883210740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55753dae6180 con 0x55753dbc2880 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f8883210740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55753da88480 con 0x55753dbc0900 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f8883210740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55753da88900 con 0x55753dbc0d80 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2880 0x55753db9bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2880 0x55753db9bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51500/0 (socket says 172.21.15.28:51500) 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 -- 172.21.15.28:0/2950451211 learned_addr learned my addr 172.21.15.28:0/2950451211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88817b7640 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc0d80 0x55753db9b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f8880fb6640 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0900 0x55753db9b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 -- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc0d80 msgr2=0x55753db9b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc0d80 0x55753db9b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 -- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0900 msgr2=0x55753db9b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0900 0x55753db9b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 -- 172.21.15.28:0/2950451211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557539a25860 con 0x55753dbc2880 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88817b7640 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc0d80 0x55753db9b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.269+0000 7f88807b5640 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2880 0x55753db9bb80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x55753e827140 tx=0x55753e911800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2f08dbf722eb38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f887c7ad640 1 -- 172.21.15.28:0/2950451211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557539a1e680 con 0x55753dbc2880 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f887c7ad640 1 -- 172.21.15.28:0/2950451211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55753da70540 con 0x55753dbc2880 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f887c7ad640 1 -- 172.21.15.28:0/2950451211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557539a1eb60 con 0x55753dbc2880 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 -- 172.21.15.28:0/2950451211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2880 msgr2=0x55753db9bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2880 0x55753db9bb80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x55753e827140 tx=0x55753e911800 comp rx=0 tx=0).stop 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 -- 172.21.15.28:0/2950451211 shutdown_connections 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2880 0x55753db9bb80 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0900 0x55753db9b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 --2- 172.21.15.28:0/2950451211 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc0d80 0x55753db9b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 -- 172.21.15.28:0/2950451211 >> 172.21.15.28:0/2950451211 conn(0x55753dbc1200 msgr2=0x55753db0c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 -- 172.21.15.28:0/2950451211 shutdown_connections 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.270+0000 7f8883210740 1 -- 172.21.15.28:0/2950451211 wait complete. 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.271+0000 7f8883210740 1 Processor -- start 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.271+0000 7f8883210740 1 -- start start 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f8883210740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0d80 0x55753db9b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f8883210740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc2880 0x55753db9b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f8883210740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2d00 0x55753db9bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f8883210740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55753dae7800 con 0x55753dbc2d00 2024-04-25T19:13:09.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f8883210740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55753dae6180 con 0x55753dbc0d80 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f8883210740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55753dae6d80 con 0x55753dbc2880 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f88807b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2d00 0x55753db9bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f88807b5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2d00 0x55753db9bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51502/0 (socket says 172.21.15.28:51502) 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.272+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 learned_addr learned my addr 172.21.15.28:0/804767416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f8880fb6640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc2880 0x55753db9b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0d80 0x55753db9b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc2880 msgr2=0x55753db9b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f88807b5640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc2880 0x55753db9b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0d80 msgr2=0x55753db9b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f88807b5640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0d80 0x55753db9b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557539a25680 con 0x55753dbc2d00 2024-04-25T19:13:09.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f8880fb6640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc2880 0x55753db9b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc0d80 0x55753db9b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f88807b5640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2d00 0x55753db9bb80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x55753e827e00 tx=0x55753db28540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f887afaa640 1 -- 172.21.15.28:0/804767416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557539a1e4e0 con 0x55753dbc2d00 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f887afaa640 1 -- 172.21.15.28:0/804767416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55753da70540 con 0x55753dbc2d00 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f887afaa640 1 -- 172.21.15.28:0/804767416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x557539a1e680 con 0x55753dbc2d00 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x557539a254a0 con 0x55753dbc2d00 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557539a252c0 con 0x55753dbc2d00 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.273+0000 7f8883210740 4 datascan.init: forcing metadata pool 'recovery_fs_metadata' 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f887afaa640 1 -- 172.21.15.28:0/804767416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x55753e94e900 con 0x55753dbc2d00 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f887afaa640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55753dbc1200 0x55753db9c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f887afaa640 1 -- 172.21.15.28:0/804767416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55753e96aa00 con 0x55753dbc2d00 2024-04-25T19:13:09.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55753dbc1200 0x55753db9c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f8883210740 4 datascan.main: data pool 'cephfs_data' has ID 6 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f8883210740 4 datascan.main: opening data pool 'cephfs_data' 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55753dbc3180 0x55753db9c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:1 7.14 7:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753e968c00 con 0x55753dbc3180 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.275+0000 7f8880fb6640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55753dbc3180 0x55753db9c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.276+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55753dbc1200 0x55753db9c100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55753db2a660 tx=0x55753db28a80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.276+0000 7f8880fb6640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55753dbc3180 0x55753db9c680 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.276+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv1 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.276+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55753dbc3600 0x55753db9cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.276+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:2 7.1f 7:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753e969000 con 0x55753dbc3600 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.276+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55753dbc3600 0x55753db9cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.277+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55753dbc3600 0x55753db9cc00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.277+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv1 ondisk = 0) ==== 162+0+16 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.277+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55753dbc3a80 0x55753db9d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.277+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:3 6.0 6:00000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969400 con 0x55753dbc3a80 2024-04-25T19:13:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.277+0000 7f8880fb6640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55753dbc3a80 0x55753db9d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.278+0000 7f8880fb6640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55753dbc3a80 0x55753db9d180 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.278+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.278+0000 7f887bfac640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55753de34000 0x55753de3c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.278+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:4 6.20 6:04000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40000 con 0x55753de34000 2024-04-25T19:13:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.278+0000 7f88807b5640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55753de34000 0x55753de3c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.279+0000 7f88807b5640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55753de34000 0x55753de3c000 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.279+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.279+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:5 6.10 6:08000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40400 con 0x55753dbc3a80 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.279+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 2 ==== osd_op_reply(5 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.279+0000 7f887bfac640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55753de34480 0x55753de3c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.279+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:6 6.30 6:0c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40800 con 0x55753de34480 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.279+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55753de34480 0x55753de3c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.280+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55753de34480 0x55753de3c580 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.280+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 1 ==== osd_op_reply(6 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.280+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:7 6.8 6:10000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41000 con 0x55753de34480 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.280+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 2 ==== osd_op_reply(7 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.281+0000 7f887bfac640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55753de34900 0x55753de3cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.281+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:8 6.28 6:14000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40c00 con 0x55753de34900 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.281+0000 7f8880fb6640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55753de34900 0x55753de3cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.281+0000 7f8880fb6640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55753de34900 0x55753de3cb00 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.281+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 1 ==== osd_op_reply(8 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.281+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:9 6.18 6:18000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41400 con 0x55753dbc3a80 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.281+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 3 ==== osd_op_reply(9 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.282+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:10 6.38 6:1c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e968c00 con 0x55753dbc3180 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.282+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.282+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:11 6.4 6:20000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40400 con 0x55753de34000 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.282+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 2 ==== osd_op_reply(11 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.282+0000 7f887bfac640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55753de34d80 0x55753de3d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.282+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:12 6.24 6:24000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969400 con 0x55753de34d80 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.282+0000 7f88807b5640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55753de34d80 0x55753de3d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.283+0000 7f88807b5640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55753de34d80 0x55753de3d080 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.283+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 1 ==== osd_op_reply(12 [pgnls-filter start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.283+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:13 6.14 6:28000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40c00 con 0x55753de34900 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.283+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 2 ==== osd_op_reply(13 [pgnls-filter start_epoch 46 out=49b] v38'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.283+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:14 6.34 6:2c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e90b400 con 0x55753dbc3a80 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.284+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 4 ==== osd_op_reply(14 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.284+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:15 6.c 6:30000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41800 con 0x55753dbc3180 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.284+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 3 ==== osd_op_reply(15 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.284+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:16 6.2c 6:34000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41c00 con 0x55753dbc3600 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.284+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 2 ==== osd_op_reply(16 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.284+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:17 6.1c 6:38000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56000 con 0x55753de34480 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.285+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 3 ==== osd_op_reply(17 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.285+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:18 6.3c 6:3c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56000 con 0x55753de34900 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.285+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 3 ==== osd_op_reply(18 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.285+0000 7f887c7ad640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55753de35b00 0x55753de3e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.285+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:19 6.2 6:40000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56400 con 0x55753de35b00 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.285+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55753de35b00 0x55753de3e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.286+0000 7f88817b7640 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55753de35b00 0x55753de3e100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.286+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 1 ==== osd_op_reply(19 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.286+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:20 6.22 6:44000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40800 con 0x55753dbc3600 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.287+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 3 ==== osd_op_reply(20 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.287+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:21 6.12 6:48000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56800 con 0x55753de34480 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.287+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 4 ==== osd_op_reply(21 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.287+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:22 6.32 6:4c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41000 con 0x55753de35b00 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.287+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 2 ==== osd_op_reply(22 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.287+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:23 6.a 6:50000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56c00 con 0x55753de34d80 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.288+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 2 ==== osd_op_reply(23 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.288+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:24 6.2a 6:54000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41c00 con 0x55753de35b00 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.288+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 3 ==== osd_op_reply(24 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.288+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:25 6.1a 6:58000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40000 con 0x55753de35b00 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.288+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 4 ==== osd_op_reply(25 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.289+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:26 6.3a 6:5c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969000 con 0x55753de34000 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.289+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 3 ==== osd_op_reply(26 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.289+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:27 6.6 6:60000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40400 con 0x55753dbc3180 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.289+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 4 ==== osd_op_reply(27 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.289+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:28 6.26 6:64000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40800 con 0x55753de34000 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.289+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 4 ==== osd_op_reply(28 [pgnls-filter start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.289+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:29 6.16 6:68000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e90b400 con 0x55753dbc3a80 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.290+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 5 ==== osd_op_reply(29 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.290+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:30 6.36 6:6c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41000 con 0x55753de34d80 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.290+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 3 ==== osd_op_reply(30 [pgnls-filter start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.290+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:31 6.e 6:70000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969400 con 0x55753de35b00 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.291+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 5 ==== osd_op_reply(31 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.291+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:32 6.2e 6:74000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41c00 con 0x55753dbc3600 2024-04-25T19:13:09.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.291+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 4 ==== osd_op_reply(32 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.291+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:33 6.1e 6:78000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e968c00 con 0x55753de35b00 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.291+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 6 ==== osd_op_reply(33 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.291+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:34 6.3e 6:7c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40000 con 0x55753dbc3180 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.292+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 5 ==== osd_op_reply(34 [pgnls-filter start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.292+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:35 6.1 6:80000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41800 con 0x55753de34000 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.292+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 5 ==== osd_op_reply(35 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.292+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:36 6.21 6:84000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41400 con 0x55753de34480 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.292+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 5 ==== osd_op_reply(36 [pgnls-filter start_epoch 46 out=49b] v38'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.292+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:37 6.11 6:88000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56000 con 0x55753dbc3180 2024-04-25T19:13:09.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.293+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 6 ==== osd_op_reply(37 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.293+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:38 6.31 6:8c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40c00 con 0x55753de34d80 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.293+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 4 ==== osd_op_reply(38 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.293+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:39 6.9 6:90000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56400 con 0x55753dbc3a80 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.293+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 6 ==== osd_op_reply(39 [pgnls-filter start_epoch 46 out=49b] v38'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.293+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:40 6.29 6:94000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57c00 con 0x55753de34480 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.294+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 6 ==== osd_op_reply(40 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.294+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:41 6.19 6:98000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56800 con 0x55753de34d80 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.294+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 5 ==== osd_op_reply(41 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.294+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:42 6.39 6:9c000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40800 con 0x55753de34d80 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.295+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 6 ==== osd_op_reply(42 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.295+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:43 6.5 6:a0000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56c00 con 0x55753de34480 2024-04-25T19:13:09.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.295+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 7 ==== osd_op_reply(43 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.295+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:44 6.25 6:a4000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41000 con 0x55753dbc3600 2024-04-25T19:13:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.295+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 5 ==== osd_op_reply(44 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.295+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:45 6.15 6:a8000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57000 con 0x55753de34480 2024-04-25T19:13:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.296+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 8 ==== osd_op_reply(45 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.296+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:46 6.35 6:ac000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41800 con 0x55753de34d80 2024-04-25T19:13:09.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.296+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 7 ==== osd_op_reply(46 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.296+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:47 6.d 6:b0000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57400 con 0x55753de34d80 2024-04-25T19:13:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.297+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 8 ==== osd_op_reply(47 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.297+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:48 6.2d 6:b4000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40c00 con 0x55753de34900 2024-04-25T19:13:09.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.297+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 4 ==== osd_op_reply(48 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.297+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:49 6.1d 6:b8000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57800 con 0x55753de34000 2024-04-25T19:13:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.297+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 6 ==== osd_op_reply(49 [pgnls-filter start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.297+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:50 6.3d 6:bc000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969000 con 0x55753dbc3a80 2024-04-25T19:13:09.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.297+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 7 ==== osd_op_reply(50 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.298+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:51 6.3 6:c0000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56000 con 0x55753de34480 2024-04-25T19:13:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.298+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 9 ==== osd_op_reply(51 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.298+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:52 6.23 6:c4000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40400 con 0x55753dbc3600 2024-04-25T19:13:09.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.298+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 6 ==== osd_op_reply(52 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.298+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:53 6.13 6:c8000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56400 con 0x55753dbc3180 2024-04-25T19:13:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.299+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 7 ==== osd_op_reply(53 [pgnls-filter start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.299+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:54 6.33 6:cc000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40000 con 0x55753de34d80 2024-04-25T19:13:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.299+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 9 ==== osd_op_reply(54 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.299+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:55 6.b 6:d0000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57c00 con 0x55753dbc3180 2024-04-25T19:13:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.299+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 8 ==== osd_op_reply(55 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.299+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:56 6.2b 6:d4000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40c00 con 0x55753de34d80 2024-04-25T19:13:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.300+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 10 ==== osd_op_reply(56 [pgnls-filter start_epoch 46 out=49b] v38'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.300+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:57 6.1b 6:d8000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56c00 con 0x55753dbc3180 2024-04-25T19:13:09.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.300+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 9 ==== osd_op_reply(57 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.300+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:58 6.3b 6:dc000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41c00 con 0x55753de34900 2024-04-25T19:13:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.300+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 5 ==== osd_op_reply(58 [pgnls-filter start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.300+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:59 6.7 6:e0000000::::head [pgnls-filter start_epoch 46 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969000 con 0x55753de34d80 2024-04-25T19:13:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.301+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 11 ==== osd_op_reply(59 [pgnls-filter start_epoch 46 out=81b] v46'10 uv10 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.301+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:60 6.0 6:00000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969800 con 0x55753dbc3a80 2024-04-25T19:13:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.301+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 8 ==== osd_op_reply(60 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.301+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:61 6.20 6:04000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969400 con 0x55753de34000 2024-04-25T19:13:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.301+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 7 ==== osd_op_reply(61 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.301+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:62 6.10 6:08000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41400 con 0x55753dbc3a80 2024-04-25T19:13:09.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.302+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 9 ==== osd_op_reply(62 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.302+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:63 6.30 6:0c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e968c00 con 0x55753de34480 2024-04-25T19:13:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.302+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 10 ==== osd_op_reply(63 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.302+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:64 6.8 6:10000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41000 con 0x55753de34480 2024-04-25T19:13:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.302+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 11 ==== osd_op_reply(64 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.302+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:65 6.28 6:14000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e90b400 con 0x55753de34900 2024-04-25T19:13:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.303+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.303+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:66 6.18 6:18000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40800 con 0x55753dbc3a80 2024-04-25T19:13:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.303+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 10 ==== osd_op_reply(66 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.303+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:67 6.38 6:1c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41c00 con 0x55753dbc3180 2024-04-25T19:13:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.303+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 10 ==== osd_op_reply(67 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.303+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:68 6.4 6:20000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41800 con 0x55753de34000 2024-04-25T19:13:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.303+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 8 ==== osd_op_reply(68 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.304+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:69 6.24 6:24000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41400 con 0x55753de34d80 2024-04-25T19:13:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.304+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 12 ==== osd_op_reply(69 [pgnls-filter start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.304+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:70 6.14 6:28000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40000 con 0x55753de34900 2024-04-25T19:13:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.304+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 7 ==== osd_op_reply(70 [pgnls-filter start_epoch 46 out=49b] v38'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.304+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:71 6.34 6:2c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40800 con 0x55753dbc3a80 2024-04-25T19:13:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.305+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 11 ==== osd_op_reply(71 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.305+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:72 6.c 6:30000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56800 con 0x55753dbc3180 2024-04-25T19:13:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.305+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 11 ==== osd_op_reply(72 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.305+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:73 6.2c 6:34000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e90b400 con 0x55753dbc3600 2024-04-25T19:13:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.305+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 7 ==== osd_op_reply(73 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.305+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:74 6.1c 6:38000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57400 con 0x55753de34480 2024-04-25T19:13:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.306+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 12 ==== osd_op_reply(74 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.306+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:75 6.3c 6:3c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969800 con 0x55753de34900 2024-04-25T19:13:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.306+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 8 ==== osd_op_reply(75 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.306+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:76 6.2 6:40000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57800 con 0x55753de35b00 2024-04-25T19:13:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.306+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 7 ==== osd_op_reply(76 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.306+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:77 6.22 6:44000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56400 con 0x55753dbc3600 2024-04-25T19:13:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.307+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 8 ==== osd_op_reply(77 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.307+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:78 6.12 6:48000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57000 con 0x55753de34480 2024-04-25T19:13:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.307+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 13 ==== osd_op_reply(78 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.307+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:79 6.32 6:4c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57c00 con 0x55753de35b00 2024-04-25T19:13:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.307+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 8 ==== osd_op_reply(79 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.307+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:80 6.a 6:50000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56000 con 0x55753de34d80 2024-04-25T19:13:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.308+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 13 ==== osd_op_reply(80 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.308+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:81 6.2a 6:54000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56c00 con 0x55753de35b00 2024-04-25T19:13:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.308+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 9 ==== osd_op_reply(81 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.308+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:82 6.1a 6:58000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57400 con 0x55753de35b00 2024-04-25T19:13:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.308+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 10 ==== osd_op_reply(82 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.308+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:83 6.3a 6:5c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de66000 con 0x55753de34000 2024-04-25T19:13:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.309+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 9 ==== osd_op_reply(83 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.309+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:84 6.6 6:60000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57800 con 0x55753dbc3180 2024-04-25T19:13:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.309+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 12 ==== osd_op_reply(84 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.309+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:85 6.26 6:64000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de66400 con 0x55753de34000 2024-04-25T19:13:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.309+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 10 ==== osd_op_reply(85 [pgnls-filter start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.309+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:86 6.16 6:68000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e90b400 con 0x55753dbc3a80 2024-04-25T19:13:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.310+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 12 ==== osd_op_reply(86 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.310+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:87 6.36 6:6c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de66800 con 0x55753de34d80 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.310+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 14 ==== osd_op_reply(87 [pgnls-filter start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.310+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:88 6.e 6:70000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40400 con 0x55753de35b00 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.310+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 11 ==== osd_op_reply(88 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.310+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:89 6.2e 6:74000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de66c00 con 0x55753dbc3600 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.311+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 9 ==== osd_op_reply(89 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.311+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:90 6.1e 6:78000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41000 con 0x55753de35b00 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.311+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 12 ==== osd_op_reply(90 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.311+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:91 6.3e 6:7c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de67000 con 0x55753dbc3180 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.312+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 13 ==== osd_op_reply(91 [pgnls-filter start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.312+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:92 6.1 6:80000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e968c00 con 0x55753de34000 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.312+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 11 ==== osd_op_reply(92 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.312+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:93 6.21 6:84000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de67400 con 0x55753de34480 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.312+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 14 ==== osd_op_reply(93 [pgnls-filter start_epoch 46 out=49b] v38'12 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.312+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:94 6.11 6:88000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56400 con 0x55753dbc3180 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.313+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 14 ==== osd_op_reply(94 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.313+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:95 6.31 6:8c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de67800 con 0x55753de34d80 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.313+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 15 ==== osd_op_reply(95 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.313+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:96 6.9 6:90000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57000 con 0x55753dbc3a80 2024-04-25T19:13:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.313+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 13 ==== osd_op_reply(96 [pgnls-filter start_epoch 46 out=49b] v38'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.313+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:97 6.29 6:94000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de67c00 con 0x55753de34480 2024-04-25T19:13:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.314+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 15 ==== osd_op_reply(97 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.314+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:98 6.19 6:98000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57c00 con 0x55753de34d80 2024-04-25T19:13:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.314+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 16 ==== osd_op_reply(98 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.314+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:99 6.39 6:9c000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de67000 con 0x55753de34d80 2024-04-25T19:13:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.314+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 17 ==== osd_op_reply(99 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.314+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:100 6.5 6:a0000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56c00 con 0x55753de34480 2024-04-25T19:13:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.315+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 16 ==== osd_op_reply(100 [pgnls-filter start_epoch 46 out=49b] v38'5 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.315+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:101 6.25 6:a4000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56800 con 0x55753dbc3600 2024-04-25T19:13:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.315+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 10 ==== osd_op_reply(101 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.315+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:102 6.15 6:a8000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40400 con 0x55753de34480 2024-04-25T19:13:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.315+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 17 ==== osd_op_reply(102 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.316+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:103 6.35 6:ac000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57800 con 0x55753de34d80 2024-04-25T19:13:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.316+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 18 ==== osd_op_reply(103 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.316+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:104 6.d 6:b0000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41000 con 0x55753de34d80 2024-04-25T19:13:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.316+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 19 ==== osd_op_reply(104 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.316+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:105 6.2d 6:b4000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56400 con 0x55753de34900 2024-04-25T19:13:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.316+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 9 ==== osd_op_reply(105 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.317+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:106 6.1d 6:b8000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40c00 con 0x55753de34000 2024-04-25T19:13:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.317+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.1 v2:172.21.15.28:6808/1898594658 12 ==== osd_op_reply(106 [pgnls-filter start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34000 2024-04-25T19:13:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.317+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:107 6.3d 6:bc000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de57400 con 0x55753dbc3a80 2024-04-25T19:13:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.317+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 14 ==== osd_op_reply(107 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.317+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:108 6.3 6:c0000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41800 con 0x55753de34480 2024-04-25T19:13:09.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.318+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 18 ==== osd_op_reply(108 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.318+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:109 6.23 6:c4000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de56000 con 0x55753dbc3600 2024-04-25T19:13:09.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.318+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 11 ==== osd_op_reply(109 [pgnls-filter start_epoch 46 out=49b] v38'7 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.318+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:110 6.13 6:c8000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41400 con 0x55753dbc3180 2024-04-25T19:13:09.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.318+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 15 ==== osd_op_reply(110 [pgnls-filter start_epoch 46 out=49b] v38'15 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.318+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:111 6.33 6:cc000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e90b400 con 0x55753de34d80 2024-04-25T19:13:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.319+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 20 ==== osd_op_reply(111 [pgnls-filter start_epoch 46 out=49b] v38'10 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.319+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:112 6.b 6:d0000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969000 con 0x55753dbc3180 2024-04-25T19:13:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.319+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 16 ==== osd_op_reply(112 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.319+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:113 6.2b 6:d4000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969800 con 0x55753de34d80 2024-04-25T19:13:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.319+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 21 ==== osd_op_reply(113 [pgnls-filter start_epoch 46 out=49b] v38'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.319+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:114 6.1b 6:d8000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753e969400 con 0x55753dbc3180 2024-04-25T19:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.320+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 17 ==== osd_op_reply(114 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.320+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:115 6.3b 6:dc000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de41c00 con 0x55753de34900 2024-04-25T19:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.320+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 10 ==== osd_op_reply(115 [pgnls-filter start_epoch 46 out=49b] v38'4 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.320+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:116 6.7 6:e0000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de66400 con 0x55753de34d80 2024-04-25T19:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.320+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 22 ==== osd_op_reply(116 [pgnls-filter start_epoch 46 out=81b] v46'10 uv10 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.320+0000 7f8883210740 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-04-25T19:13:09.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.320+0000 7f8883210740 10 datascan.operator(): handling object 10000000001.0 2024-04-25T19:13:09.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:117 6.7 6:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753de78000 con 0x55753de34d80 2024-04-25T19:13:09.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 <== osd.5 v2:172.21.15.53:6800/2342659583 23 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (95) out=95b,getxattr (30) out=30b,getxattr] v0'0 uv10 ondisk = 0) ==== 416+0+163 (crc 0 0 0) 0x55753db10780 con 0x55753de34d80 2024-04-25T19:13:09.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 20 datascan.operator(): loaded layout from xattr: pi: 6 os: 4194304 sc: 1 su: 4194304 2024-04-25T19:13:09.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-04-25T19:13:09.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/sixmegs 2024-04-25T19:13:09.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=sixmegs 2024-04-25T19:13:09.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:118 7.10 7:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753de78400 con 0x55753dbc3180 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 18 ==== osd_op_reply(118 10000000000.00000000 [getxattr] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 164+0+0 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 4 datascan.get_frag_of: Unexpected error on '10000000000.00000000': (2) No such file or directory 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 20 datascan.inject_with_backtrace: don't know fragment for 0x0x10000000000/sixmegs, will insert to root 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:119 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753de78800 con 0x55753dbc3180 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 19 ==== osd_op_reply(119 10000000000.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 164+0+0 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 10 datascan.find_or_create_dirfrag: read_version = 0 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.321+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:120 7.10 7:0cf6ad0f:::10000000000.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55753de78c00 con 0x55753dbc3180 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.323+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 20 ==== osd_op_reply(120 10000000000.00000000 [create,omap-set-header] v46'1 uv1 ondisk = 0) ==== 206+0+0 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.323+0000 7f8883210740 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x10000000000 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.323+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:121 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753e969000 con 0x55753dbc3180 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.323+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 21 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv1 ondisk = 0) ==== 164+0+4 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.323+0000 7f8883210740 20 datascan.read_dentry: oid=10000000000.00000000 dname=sixmegs frag=*, r=0 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.323+0000 7f8883210740 20 datascan.read_dentry: sixmegs_head not found in result 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.323+0000 7f8883210740 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/sixmegs with size=6291456 bytes 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.323+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:122 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55753e969400 con 0x55753dbc3180 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.324+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 22 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v46'2 uv2 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.324+0000 7f8883210740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/sixmegs pointing to 0x0x10000000001 2024-04-25T19:13:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.324+0000 7f8883210740 10 datascan.inject_with_backtrace: backptr: 0x0x1/subdir 2024-04-25T19:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.324+0000 7f8883210740 20 datascan.get_frag_of: dirino=0x1 target_dname=subdir 2024-04-25T19:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.324+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:123 7.1f 7:ff5b34d6:::1.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753de57000 con 0x55753dbc3600 2024-04-25T19:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.324+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 12 ==== osd_op_reply(123 1.00000000 [getxattr] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 154+0+0 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.325+0000 7f8883210740 4 datascan.get_frag_of: Unexpected error on '1.00000000': (2) No such file or directory 2024-04-25T19:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.325+0000 7f8883210740 20 datascan.inject_with_backtrace: don't know fragment for 0x0x1/subdir, will insert to root 2024-04-25T19:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.325+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:124 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753de56400 con 0x55753dbc3600 2024-04-25T19:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.325+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 13 ==== osd_op_reply(124 1.00000000 [omap-get-header] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 154+0+0 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.325+0000 7f8883210740 10 datascan.find_or_create_dirfrag: read_version = 0 2024-04-25T19:13:09.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.325+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:125 7.1f 7:ff5b34d6:::1.00000000:head [create,omap-set-header in=274b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55753de57400 con 0x55753dbc3600 2024-04-25T19:13:09.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.326+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 14 ==== osd_op_reply(125 1.00000000 [create,omap-set-header] v46'2 uv2 ondisk = 0) ==== 196+0+0 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.326+0000 7f8883210740 10 datascan.find_or_create_dirfrag: Created dirfrag: 0x0x1 2024-04-25T19:13:09.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.326+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:126 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55753de78400 con 0x55753dbc3600 2024-04-25T19:13:09.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.327+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 15 ==== osd_op_reply(126 1.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv2 ondisk = 0) ==== 154+0+4 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.327+0000 7f8883210740 20 datascan.read_dentry: oid=1.00000000 dname=subdir frag=*, r=0 2024-04-25T19:13:09.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.327+0000 7f8883210740 20 datascan.read_dentry: subdir_head not found in result 2024-04-25T19:13:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.327+0000 7f8883210740 10 datascan.inject_with_backtrace: Linking ancestor directory of inode 0x0x10000000000 at 0x0x1:subdir 2024-04-25T19:13:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.327+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:127 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55753de78800 con 0x55753dbc3600 2024-04-25T19:13:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.328+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 16 ==== osd_op_reply(127 1.00000000 [omap-set-vals] v46'3 uv3 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.328+0000 7f8883210740 20 datascan.inject_linkage: Injected dentry 0x0x1/subdir pointing to 0x0x10000000000 2024-04-25T19:13:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.328+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:128 6.27 6:e4000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de78c00 con 0x55753dbc3600 2024-04-25T19:13:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.329+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.6 v2:172.21.15.53:6809/3653668189 17 ==== osd_op_reply(128 [pgnls-filter start_epoch 46 out=49b] v38'2 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753dbc3600 2024-04-25T19:13:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.329+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:129 6.17 6:e8000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de66800 con 0x55753de35b00 2024-04-25T19:13:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.329+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.4 v2:172.21.15.53:6824/2055146181 13 ==== osd_op_reply(129 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de35b00 2024-04-25T19:13:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.329+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:130 6.37 6:ec000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40000 con 0x55753dbc3a80 2024-04-25T19:13:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.329+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.0 v2:172.21.15.28:6800/1137456103 15 ==== osd_op_reply(130 [pgnls-filter start_epoch 46 out=49b] v38'11 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3a80 2024-04-25T19:13:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.329+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:131 6.f 6:f0000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de67800 con 0x55753de34900 2024-04-25T19:13:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.330+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.2 v2:172.21.15.28:6816/3410596961 11 ==== osd_op_reply(131 [pgnls-filter start_epoch 46 out=49b] v38'9 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753de34900 2024-04-25T19:13:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.330+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:132 6.2f 6:f4000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de40800 con 0x55753de34480 2024-04-25T19:13:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.330+0000 7f88817b7640 1 -- 172.21.15.28:0/804767416 <== osd.7 v2:172.21.15.53:6808/79451145 19 ==== osd_op_reply(132 [pgnls-filter start_epoch 46 out=49b] v38'9 uv1 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10500 con 0x55753de34480 2024-04-25T19:13:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.330+0000 7f887bfac640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:133 6.1f 6:f8000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de67000 con 0x55753dbc3180 2024-04-25T19:13:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.330+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 23 ==== osd_op_reply(133 [pgnls-filter start_epoch 46 out=49b] v38'6 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.330+0000 7f887c7ad640 1 -- 172.21.15.28:0/804767416 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:134 6.3f 6:fc000000::::head [pgnls-filter start_epoch 46 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55753de78400 con 0x55753dbc3180 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8880fb6640 1 -- 172.21.15.28:0/804767416 <== osd.3 v2:172.21.15.28:6818/1928233777 24 ==== osd_op_reply(134 [pgnls-filter start_epoch 46 out=49b] v38'8 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55753db10000 con 0x55753dbc3180 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55753dbc3a80 msgr2=0x55753db9d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55753dbc3a80 0x55753db9d180 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55753de34000 msgr2=0x55753de3c000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55753de34000 0x55753de3c000 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55753de34900 msgr2=0x55753de3cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55753de34900 0x55753de3cb00 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55753dbc3180 msgr2=0x55753db9c680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55753dbc3180 0x55753db9c680 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55753de35b00 msgr2=0x55753de3e100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55753de35b00 0x55753de3e100 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55753de34d80 msgr2=0x55753de3d080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55753de34d80 0x55753de3d080 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55753dbc3600 msgr2=0x55753db9cc00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55753dbc3600 0x55753db9cc00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55753de34480 msgr2=0x55753de3c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55753de34480 0x55753de3c580 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 reap_dead start 2024-04-25T19:13:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 reap_dead start 2024-04-25T19:13:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 reap_dead start 2024-04-25T19:13:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f88807b5640 1 -- 172.21.15.28:0/804767416 reap_dead start 2024-04-25T19:13:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55753dbc1200 msgr2=0x55753db9c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55753dbc1200 0x55753db9c100 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55753db2a660 tx=0x55753db28a80 comp rx=0 tx=0).stop 2024-04-25T19:13:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2d00 msgr2=0x55753db9bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2d00 0x55753db9bb80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x55753e827e00 tx=0x55753db28540 comp rx=0 tx=0).stop 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 shutdown_connections 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55753dbc1200 0x55753db9c100 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 --2- 172.21.15.28:0/804767416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc2d00 0x55753db9bb80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 >> 172.21.15.28:0/804767416 conn(0x55753dbc0900 msgr2=0x55753db0c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 shutdown_connections 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.331+0000 7f8883210740 1 -- 172.21.15.28:0/804767416 wait complete. 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 -- 172.21.15.28:0/2931074542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc1680 msgr2=0x55753db9a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc1680 0x55753db9a000 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x55753e845cb0 tx=0x55753db2c1e0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 -- 172.21.15.28:0/2931074542 shutdown_connections 2024-04-25T19:13:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55753dbc1f80 0x55753db9ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55753dbc1b00 0x55753db9a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 --2- 172.21.15.28:0/2931074542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55753dbc1680 0x55753db9a000 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 -- 172.21.15.28:0/2931074542 >> 172.21.15.28:0/2931074542 conn(0x55753dbc2400 msgr2=0x55753db0d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 -- 172.21.15.28:0/2931074542 shutdown_connections 2024-04-25T19:13:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.332+0000 7f8883210740 1 -- 172.21.15.28:0/2931074542 wait complete. 2024-04-25T19:13:09.392 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--alternate-pool', 'recovery_fs_metadata', '--filesystem', 'cephfs', '--force-corrupt', '--force-init', 'cephfs_data'] in time 0:00:00.185829, result: 2024-04-25T19:13:09.392 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links --filesystem recovery_fs 2024-04-25T19:13:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.442+0000 7fcb74f22740 1 -- start start 2024-04-25T19:13:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.442+0000 7fcb74f22740 1 Processor -- start 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652d680 0x55c3c6446000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652db00 0x55c3c6446580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652df80 0x55c3c6446b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb74f22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3c6452600 con 0x55c3c652db00 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb74f22740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c3c6452480 con 0x55c3c652df80 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb74f22740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55c3c6452300 con 0x55c3c652d680 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652db00 0x55c3c6446580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652db00 0x55c3c6446580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51520/0 (socket says 172.21.15.28:51520) 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 -- 172.21.15.28:0/1849568254 learned_addr learned my addr 172.21.15.28:0/1849568254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652d680 0x55c3c6446000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7368e640 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652df80 0x55c3c6446b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 -- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652d680 msgr2=0x55c3c6446000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652d680 0x55c3c6446000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 -- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652df80 msgr2=0x55c3c6446b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652df80 0x55c3c6446b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 -- 172.21.15.28:0/1849568254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3c2391680 con 0x55c3c652db00 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652d680 0x55c3c6446000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7368e640 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652df80 0x55c3c6446b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.444+0000 7fcb7268c640 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652db00 0x55c3c6446580 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x55c3c71b7cb0 tx=0x55c3c6494600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819ef9f451de6d08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb70688640 1 -- 172.21.15.28:0/1849568254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c3c238a680 con 0x55c3c652db00 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb70688640 1 -- 172.21.15.28:0/1849568254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55c3c63dcc40 con 0x55c3c652db00 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb70688640 1 -- 172.21.15.28:0/1849568254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c3c238ab60 con 0x55c3c652db00 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb74f22740 1 -- 172.21.15.28:0/1849568254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3c23914a0 con 0x55c3c652db00 2024-04-25T19:13:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb74f22740 1 -- 172.21.15.28:0/1849568254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55c3c2391a40 con 0x55c3c652db00 2024-04-25T19:13:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb74f22740 4 waiting for MDS map... 2024-04-25T19:13:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb70688640 1 -- 172.21.15.28:0/1849568254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55c3c726ea00 con 0x55c3c652db00 2024-04-25T19:13:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb70688640 1 -- 172.21.15.28:0/1849568254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x55c3c63ed180 con 0x55c3c652db00 2024-04-25T19:13:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb74f22740 4 Got MDS map 25 2024-04-25T19:13:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb74f22740 4 datascan.main: Using metadata pool output 2024-04-25T19:13:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.445+0000 7fcb74f22740 4 datascan.main: connecting to RADOS... 2024-04-25T19:13:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.446+0000 7fcb74f22740 1 Processor -- start 2024-04-25T19:13:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.446+0000 7fcb74f22740 1 -- start start 2024-04-25T19:13:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652c900 0x55c3c6447600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb74f22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3c6452180 con 0x55c3c652cd80 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb74f22740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c3c63f4480 con 0x55c3c652e880 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb74f22740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55c3c63f4900 con 0x55c3c652c900 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb7368e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb7368e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51522/0 (socket says 172.21.15.28:51522) 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb7368e640 1 -- 172.21.15.28:0/421202728 learned_addr learned my addr 172.21.15.28:0/421202728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb7268c640 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.447+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652c900 0x55c3c6447600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb7368e640 1 -- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652c900 msgr2=0x55c3c6447600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb7368e640 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652c900 0x55c3c6447600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb7368e640 1 -- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 msgr2=0x55c3c6447b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb7368e640 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb7368e640 1 -- 172.21.15.28:0/421202728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3c2391860 con 0x55c3c652cd80 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb7268c640 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb7368e640 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447080 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x55c3c727d200 tx=0x55c3c6494c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1599658bbd5021e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb6e684640 1 -- 172.21.15.28:0/421202728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c3c238a9c0 con 0x55c3c652cd80 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb6e684640 1 -- 172.21.15.28:0/421202728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55c3c63dca80 con 0x55c3c652cd80 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb6e684640 1 -- 172.21.15.28:0/421202728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c3c238a680 con 0x55c3c652cd80 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb74f22740 1 -- 172.21.15.28:0/421202728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 msgr2=0x55c3c6447080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb74f22740 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447080 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x55c3c727d200 tx=0x55c3c6494c60 comp rx=0 tx=0).stop 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb74f22740 1 -- 172.21.15.28:0/421202728 shutdown_connections 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb74f22740 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb74f22740 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652c900 0x55c3c6447600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb74f22740 1 --2- 172.21.15.28:0/421202728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447080 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.448+0000 7fcb74f22740 1 -- 172.21.15.28:0/421202728 >> 172.21.15.28:0/421202728 conn(0x55c3c652d200 msgr2=0x55c3c6478000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.449+0000 7fcb74f22740 1 -- 172.21.15.28:0/421202728 shutdown_connections 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.449+0000 7fcb74f22740 1 -- 172.21.15.28:0/421202728 wait complete. 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.449+0000 7fcb74f22740 1 Processor -- start 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.449+0000 7fcb74f22740 1 -- start start 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.450+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb74f22740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652ed00 0x55c3c6447b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb74f22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c3c6453980 con 0x55c3c652cd80 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb74f22740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55c3c6452180 con 0x55c3c652e880 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb74f22740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55c3c6453800 con 0x55c3c652ed00 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51532/0 (socket says 172.21.15.28:51532) 2024-04-25T19:13:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 learned_addr learned my addr 172.21.15.28:0/2852476800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652ed00 0x55c3c6447b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652ed00 msgr2=0x55c3c6447b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652ed00 0x55c3c6447b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 msgr2=0x55c3c6447080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3c2391680 con 0x55c3c652cd80 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652ed00 0x55c3c6447b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652e880 0x55c3c6447080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447600 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x55c3c727dec0 tx=0x55c3c6494720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb6ce81640 1 -- 172.21.15.28:0/2852476800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c3c238ab60 con 0x55c3c652cd80 2024-04-25T19:13:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb6ce81640 1 -- 172.21.15.28:0/2852476800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55c3c63dca80 con 0x55c3c652cd80 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb6ce81640 1 -- 172.21.15.28:0/2852476800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55c3c238a4e0 con 0x55c3c652cd80 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c3c23914a0 con 0x55c3c652cd80 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3c23912c0 con 0x55c3c652cd80 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.451+0000 7fcb74f22740 4 datascan.init: resolving metadata pool 7 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb6ce81640 1 -- 172.21.15.28:0/2852476800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x55c3c72c2900 con 0x55c3c652cd80 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb6ce81640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55c3c652d200 0x55c3c6448100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb6ce81640 1 -- 172.21.15.28:0/2852476800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55c3c72e6600 con 0x55c3c652cd80 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb74f22740 4 datascan.init: found metadata pool 'recovery_fs_metadata' 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb74f22740 4 datascan.main: resolving metadata pool 7 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55c3c652d200 0x55c3c6448100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55c3c652f180 0x55c3c6448680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:1 7.0 7:00000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c652f180 2024-04-25T19:13:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.453+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55c3c652f180 0x55c3c6448680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.454+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55c3c652d200 0x55c3c6448100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55c3c6497200 tx=0x55c3c670cae0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.454+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55c3c652f180 0x55c3c6448680 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.454+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.5 v2:172.21.15.53:6800/2342659583 1 ==== osd_op_reply(1 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c652f180 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.454+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55c3c652fa80 0x55c3c6794000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.454+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:2 7.10 7:08000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c652fa80 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.454+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55c3c652fa80 0x55c3c6794000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.455+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55c3c652fa80 0x55c3c6794000 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.455+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== osd_op_reply(2 [pgnls start_epoch 46 out=81b] v46'2 uv2 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.455+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55c3c67a0000 0x55c3c6794580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.455+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:3 7.8 7:10000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0000 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.455+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55c3c67a0000 0x55c3c6794580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.456+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55c3c67a0000 0x55c3c6794580 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.456+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.2 v2:172.21.15.28:6816/3410596961 1 ==== osd_op_reply(3 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0000 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.456+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55c3c67a0480 0x55c3c6794b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.456+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:4 7.18 7:18000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0480 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.456+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55c3c67a0480 0x55c3c6794b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.456+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55c3c67a0480 0x55c3c6794b00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.457+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== osd_op_reply(4 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.457+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:5 7.4 7:20000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0480 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.457+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 2 ==== osd_op_reply(5 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.457+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:6 7.14 7:28000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c652fa80 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.457+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 2 ==== osd_op_reply(6 [pgnls start_epoch 46 out=77b] v46'1 uv1 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.457+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55c3c67a0900 0x55c3c6795080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.457+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:7 7.c 7:30000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0900 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.458+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55c3c67a0900 0x55c3c6795080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.458+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55c3c67a0900 0x55c3c6795080 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.458+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== osd_op_reply(7 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.458+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:8 7.1c 7:38000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c652f180 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.459+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.5 v2:172.21.15.53:6800/2342659583 2 ==== osd_op_reply(8 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c652f180 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.459+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:9 7.2 7:40000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c652fa80 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.459+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 3 ==== osd_op_reply(9 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.459+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:10 7.12 7:48000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0900 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.459+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 2 ==== osd_op_reply(10 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.459+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:11 7.a 7:50000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c652fa80 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.460+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 4 ==== osd_op_reply(11 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.460+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55c3c67a0d80 0x55c3c6795600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.460+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:12 7.1a 7:58000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0d80 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.460+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55c3c67a0d80 0x55c3c6795600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.460+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55c3c67a0d80 0x55c3c6795600 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.461+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.4 v2:172.21.15.53:6824/2055146181 1 ==== osd_op_reply(12 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0d80 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.461+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:13 7.6 7:60000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0480 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.461+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 3 ==== osd_op_reply(13 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.461+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:14 7.16 7:68000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0d80 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.461+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.4 v2:172.21.15.53:6824/2055146181 2 ==== osd_op_reply(14 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0d80 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.461+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:15 7.e 7:70000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0000 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.462+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.2 v2:172.21.15.28:6816/3410596961 2 ==== osd_op_reply(15 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0000 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.462+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55c3c67a1200 0x55c3c6795b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.462+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:16 7.1e 7:78000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a1200 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.462+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55c3c67a1200 0x55c3c6795b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.462+0000 7fcb7268c640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55c3c67a1200 0x55c3c6795b80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.462+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.1 v2:172.21.15.28:6808/1898594658 1 ==== osd_op_reply(16 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a1200 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.462+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:17 7.1 7:80000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0d80 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.463+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.4 v2:172.21.15.53:6824/2055146181 3 ==== osd_op_reply(17 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0d80 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.463+0000 7fcb72e8d640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55c3c67a1680 0x55c3c6796100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.463+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:18 7.11 7:88000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a1680 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.463+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55c3c67a1680 0x55c3c6796100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.463+0000 7fcb7368e640 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55c3c67a1680 0x55c3c6796100 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.464+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.7 v2:172.21.15.53:6808/79451145 1 ==== osd_op_reply(18 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a1680 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.464+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:19 7.9 7:90000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0480 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.464+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 4 ==== osd_op_reply(19 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.464+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:20 7.19 7:98000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0900 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.464+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 3 ==== osd_op_reply(20 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.464+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:21 7.5 7:a0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a1680 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.465+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.7 v2:172.21.15.53:6808/79451145 2 ==== osd_op_reply(21 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a1680 2024-04-25T19:13:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.465+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:22 7.15 7:a8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a1680 2024-04-25T19:13:09.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.465+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.7 v2:172.21.15.53:6808/79451145 3 ==== osd_op_reply(22 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a1680 2024-04-25T19:13:09.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.465+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:23 7.d 7:b0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0900 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.465+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 4 ==== osd_op_reply(23 [pgnls start_epoch 46 out=74b] v46'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.466+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:24 7.1d 7:b8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a1200 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.466+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.1 v2:172.21.15.28:6808/1898594658 2 ==== osd_op_reply(24 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a1200 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.466+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:25 7.3 7:c0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c652fa80 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.466+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 5 ==== osd_op_reply(25 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.466+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:26 7.13 7:c8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0480 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.467+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 5 ==== osd_op_reply(26 [pgnls start_epoch 46 out=98b] v46'2 uv2 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.467+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:27 7.b 7:d0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a1200 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.467+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.1 v2:172.21.15.28:6808/1898594658 3 ==== osd_op_reply(27 [pgnls start_epoch 46 out=76b] v46'1 uv1 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a1200 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.467+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:28 7.1b 7:d8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0480 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.467+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 6 ==== osd_op_reply(28 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.467+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:29 7.7 7:e0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0900 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.468+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 5 ==== osd_op_reply(29 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.468+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:30 7.17 7:e8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a1680 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.468+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.7 v2:172.21.15.53:6808/79451145 4 ==== osd_op_reply(30 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a1680 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.468+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:31 7.f 7:f0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0480 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.468+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 7 ==== osd_op_reply(31 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.468+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:32 7.1f 7:f8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0900 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 6 ==== osd_op_reply(32 [pgnls start_epoch 46 out=101b] v46'3 uv3 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:33 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9400 con 0x55c3c652fa80 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 6 ==== osd_op_reply(33 10000000000.00000000 [omap-get-vals out=509b] v0'0 uv2 ondisk = 0) ==== 164+0+509 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 20 datascan.scan_links: injected ino detected 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-04-25T19:13:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 10 datascan.scan_links: step 1: handling object 100.00000000 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:34 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be000 con 0x55c3c67a0480 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.469+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 8 ==== osd_op_reply(34 100.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb74f22740 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb74f22740 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb74f22740 10 datascan.scan_links: step 1: handling object 1.00000000 2024-04-25T19:13:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:35 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be400 con 0x55c3c67a0900 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 7 ==== osd_op_reply(35 1.00000000 [omap-get-vals out=508b] v0'0 uv3 ondisk = 0) ==== 154+0+508 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb74f22740 20 datascan.scan_links: injected ino detected 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:36 7.0 7:00000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c652f180 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.5 v2:172.21.15.53:6800/2342659583 3 ==== osd_op_reply(36 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c652f180 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.470+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:37 7.10 7:08000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c652fa80 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.471+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 7 ==== osd_op_reply(37 [pgnls start_epoch 46 out=81b] v46'2 uv2 ondisk = 1) ==== 144+0+81 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.471+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:38 7.8 7:10000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0000 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.471+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.2 v2:172.21.15.28:6816/3410596961 3 ==== osd_op_reply(38 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0000 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.471+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:39 7.18 7:18000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0480 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.471+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 9 ==== osd_op_reply(39 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.471+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:40 7.4 7:20000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0480 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.472+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 10 ==== osd_op_reply(40 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.472+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:41 7.14 7:28000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c652fa80 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.472+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 8 ==== osd_op_reply(41 [pgnls start_epoch 46 out=77b] v46'1 uv1 ondisk = 1) ==== 144+0+77 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.472+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:42 7.c 7:30000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0900 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.472+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 8 ==== osd_op_reply(42 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.472+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- osd_op(unknown.0.0:43 7.1c 7:38000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c652f180 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.473+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.5 v2:172.21.15.53:6800/2342659583 4 ==== osd_op_reply(43 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c652f180 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.473+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:44 7.2 7:40000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c652fa80 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.473+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 9 ==== osd_op_reply(44 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.473+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:45 7.12 7:48000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0900 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.473+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 9 ==== osd_op_reply(45 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.473+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:46 7.a 7:50000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c652fa80 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.473+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 10 ==== osd_op_reply(46 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.473+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:47 7.1a 7:58000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0d80 2024-04-25T19:13:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.474+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.4 v2:172.21.15.53:6824/2055146181 4 ==== osd_op_reply(47 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0d80 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.474+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:48 7.6 7:60000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0480 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.474+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 11 ==== osd_op_reply(48 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.474+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:49 7.16 7:68000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0d80 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.474+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.4 v2:172.21.15.53:6824/2055146181 5 ==== osd_op_reply(49 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0d80 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.474+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- osd_op(unknown.0.0:50 7.e 7:70000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0000 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.475+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.2 v2:172.21.15.28:6816/3410596961 4 ==== osd_op_reply(50 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0000 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.475+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:51 7.1e 7:78000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a1200 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.475+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.1 v2:172.21.15.28:6808/1898594658 4 ==== osd_op_reply(51 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a1200 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.475+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:52 7.1 7:80000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0d80 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.475+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 <== osd.4 v2:172.21.15.53:6824/2055146181 6 ==== osd_op_reply(52 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c780 con 0x55c3c67a0d80 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.475+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:53 7.11 7:88000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a1680 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.476+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.7 v2:172.21.15.53:6808/79451145 5 ==== osd_op_reply(53 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a1680 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.476+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:54 7.9 7:90000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0480 2024-04-25T19:13:09.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.476+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 12 ==== osd_op_reply(54 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.476+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:55 7.19 7:98000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0900 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.476+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 10 ==== osd_op_reply(55 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.476+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:56 7.5 7:a0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a1680 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.477+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.7 v2:172.21.15.53:6808/79451145 6 ==== osd_op_reply(56 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a1680 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.477+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:57 7.15 7:a8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a1680 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.477+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.7 v2:172.21.15.53:6808/79451145 7 ==== osd_op_reply(57 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a1680 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.477+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:58 7.d 7:b0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0900 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.477+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 11 ==== osd_op_reply(58 [pgnls start_epoch 46 out=74b] v46'1 uv1 ondisk = 1) ==== 144+0+74 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.477+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:59 7.1d 7:b8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a1200 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.478+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.1 v2:172.21.15.28:6808/1898594658 5 ==== osd_op_reply(59 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a1200 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.478+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:60 7.3 7:c0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c652fa80 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.478+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 11 ==== osd_op_reply(60 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.478+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:61 7.13 7:c8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0480 2024-04-25T19:13:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.478+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 13 ==== osd_op_reply(61 [pgnls start_epoch 46 out=98b] v46'2 uv2 ondisk = 1) ==== 144+0+98 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.478+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:62 7.b 7:d0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a1200 2024-04-25T19:13:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.479+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.1 v2:172.21.15.28:6808/1898594658 6 ==== osd_op_reply(62 [pgnls start_epoch 46 out=76b] v46'1 uv1 ondisk = 1) ==== 144+0+76 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a1200 2024-04-25T19:13:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.479+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:63 7.1b 7:d8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0480 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.479+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 14 ==== osd_op_reply(63 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.479+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:64 7.7 7:e0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0900 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.479+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 12 ==== osd_op_reply(64 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.479+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- osd_op(unknown.0.0:65 7.17 7:e8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a1680 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.480+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.7 v2:172.21.15.53:6808/79451145 8 ==== osd_op_reply(65 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a1680 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.480+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:66 7.f 7:f0000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0480 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.480+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 15 ==== osd_op_reply(66 [pgnls start_epoch 46 out=49b] v0'0 uv0 ondisk = 1) ==== 144+0+49 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.480+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:67 7.1f 7:f8000000::::head [pgnls start_epoch 46 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be800 con 0x55c3c67a0900 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.480+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 13 ==== osd_op_reply(67 [pgnls start_epoch 46 out=101b] v46'3 uv3 ondisk = 1) ==== 144+0+101 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.480+0000 7fcb74f22740 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.480+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:68 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e8c00 con 0x55c3c652fa80 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 12 ==== osd_op_reply(68 10000000000.00000000 [omap-get-vals out=509b] v0'0 uv2 ondisk = 0) ==== 164+0+509 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 20 datascan.scan_links: injected ino detected 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-04-25T19:13:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: step 2: handling object 100.00000000 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:69 7.13 7:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e8800 con 0x55c3c67a0480 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 16 ==== osd_op_reply(69 100.00000000 [omap-get-vals out=5b] v0'0 uv1 ondisk = 0) ==== 156+0+5 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-04-25T19:13:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 10 datascan.scan_links: step 2: handling object 1.00000000 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.481+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:70 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e8400 con 0x55c3c67a0900 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 14 ==== osd_op_reply(70 1.00000000 [omap-get-vals out=508b] v0'0 uv3 ondisk = 0) ==== 154+0+508 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 20 datascan.scan_links: injected ino detected 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 20 datascan.scan_links: adding 0x10000000000 for future processing to fix dnfirst 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 10 datascan.scan_links: removing dup dentries from 0 objects 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 10 datascan.scan_links: processing 2 injected_inos 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 10 datascan.scan_links: handling injected_ino 0x10000000000 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x1:subdir 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:71 7.1f 7:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=19b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9000 con 0x55c3c67a0900 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 15 ==== osd_op_reply(71 1.00000000 [omap-get-vals-by-keys out=507b] v0'0 uv3 ondisk = 0) ==== 154+0+507 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 20 datascan.read_dentry: oid=1.00000000 dname=subdir frag=*, r=0 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 20 datascan.scan_links: first is now 2 2024-04-25T19:13:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.482+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:72 7.1f 7:ff5b34d6:::1.00000000:head [omap-set-vals in=507b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67be400 con 0x55c3c67a0900 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 16 ==== osd_op_reply(72 1.00000000 [omap-set-vals] v46'4 uv4 ondisk = 0) ==== 154+0+0 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c67a0900 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb74f22740 20 datascan.inject_linkage: Injected dentry 0x0x1/subdir pointing to 0x0x10000000000 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb74f22740 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb74f22740 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:sixmegs 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:73 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67bec00 con 0x55c3c652fa80 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 13 ==== osd_op_reply(73 10000000000.00000000 [omap-get-vals-by-keys out=508b] v0'0 uv2 ondisk = 0) ==== 164+0+508 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb74f22740 20 datascan.read_dentry: oid=10000000000.00000000 dname=sixmegs frag=*, r=0 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb74f22740 20 datascan.scan_links: first is now 2 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.484+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:74 7.10 7:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=508b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67bf000 con 0x55c3c652fa80 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.485+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.3 v2:172.21.15.28:6818/1928233777 14 ==== osd_op_reply(74 10000000000.00000000 [omap-set-vals] v46'3 uv3 ondisk = 0) ==== 164+0+0 (crc 0 0 0) 0x55c3c647c500 con 0x55c3c652fa80 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.485+0000 7fcb74f22740 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/sixmegs pointing to 0x0x10000000001 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.485+0000 7fcb74f22740 10 datascan.scan_links: updating inotable 2024-04-25T19:13:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.485+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:75 7.13 7:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c67bf400 con 0x55c3c67a0480 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.486+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 17 ==== osd_op_reply(75 mds0_inotable [read 0~34 out=34b] v0'0 uv2 ondisk = 0) ==== 157+0+34 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.486+0000 7fcb74f22740 10 mds.0.inotable: skip_inos was [0x10000000000~0x10000000000] 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.486+0000 7fcb74f22740 10 mds.0.inotable: skip_inos now [0x10000000002~0xfffffffffe] 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.486+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:76 7.13 7:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e9400 con 0x55c3c67a0480 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.487+0000 7fcb7268c640 1 -- 172.21.15.28:0/2852476800 <== osd.0 v2:172.21.15.28:6800/1137456103 18 ==== osd_op_reply(76 mds0_inotable [writefull 0~34] v46'3 uv3 ondisk = 0) ==== 157+0+0 (crc 0 0 0) 0x55c3c647c000 con 0x55c3c67a0480 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.487+0000 7fcb74f22740 10 datascan.scan_links: updating snaptable 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.487+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:77 7.d 7:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55c3c72e8800 con 0x55c3c67a0900 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb7368e640 1 -- 172.21.15.28:0/2852476800 <== osd.6 v2:172.21.15.53:6809/3653668189 17 ==== osd_op_reply(77 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) ==== 157+0+70 (crc 0 0 0) 0x55c3c647c280 con 0x55c3c67a0900 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55c3c67a0480 msgr2=0x55c3c6794b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55c3c67a0480 0x55c3c6794b00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55c3c67a1200 msgr2=0x55c3c6795b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55c3c67a1200 0x55c3c6795b80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55c3c67a0000 msgr2=0x55c3c6794580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x55c3c67a0000 0x55c3c6794580 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55c3c652fa80 msgr2=0x55c3c6794000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55c3c652fa80 0x55c3c6794000 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55c3c67a0d80 msgr2=0x55c3c6795600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55c3c67a0d80 0x55c3c6795600 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55c3c652f180 msgr2=0x55c3c6448680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x55c3c652f180 0x55c3c6448680 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 reap_dead start 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55c3c67a0900 msgr2=0x55c3c6795080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55c3c67a0900 0x55c3c6795080 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55c3c67a1680 msgr2=0x55c3c6796100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x55c3c67a1680 0x55c3c6796100 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 reap_dead start 2024-04-25T19:13:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 reap_dead start 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb72e8d640 1 -- 172.21.15.28:0/2852476800 reap_dead start 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55c3c652d200 msgr2=0x55c3c6448100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55c3c652d200 0x55c3c6448100 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55c3c6497200 tx=0x55c3c670cae0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 msgr2=0x55c3c6447600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447600 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x55c3c727dec0 tx=0x55c3c6494720 comp rx=0 tx=0).stop 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 shutdown_connections 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55c3c652d200 0x55c3c6448100 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 --2- 172.21.15.28:0/2852476800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652cd80 0x55c3c6447600 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 >> 172.21.15.28:0/2852476800 conn(0x55c3c652c900 msgr2=0x55c3c6478000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 shutdown_connections 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.488+0000 7fcb74f22740 1 -- 172.21.15.28:0/2852476800 wait complete. 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 -- 172.21.15.28:0/1849568254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652db00 msgr2=0x55c3c6446580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652db00 0x55c3c6446580 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x55c3c71b7cb0 tx=0x55c3c6494600 comp rx=0 tx=0).stop 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 -- 172.21.15.28:0/1849568254 shutdown_connections 2024-04-25T19:13:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c3c652df80 0x55c3c6446b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c3c652db00 0x55c3c6446580 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 --2- 172.21.15.28:0/1849568254 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55c3c652d680 0x55c3c6446000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 -- 172.21.15.28:0/1849568254 >> 172.21.15.28:0/1849568254 conn(0x55c3c652e400 msgr2=0x55c3c6479000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 -- 172.21.15.28:0/1849568254 shutdown_connections 2024-04-25T19:13:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.489+0000 7fcb74f22740 1 -- 172.21.15.28:0/1849568254 wait complete. 2024-04-25T19:13:09.501 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links', '--filesystem', 'recovery_fs'] in time 0:00:00.108857, result: 2024-04-25T19:13:09.501 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank recovery_fs:0 event recover_dentries list --alternate-pool recovery_fs_metadata 2024-04-25T19:13:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.555+0000 7f42c5520280 1 -- start start 2024-04-25T19:13:09.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.555+0000 7f42c5520280 1 Processor -- start 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c5520280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212909680 0x55d212910b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c5520280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212909b00 0x55d212910580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c5520280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d21290a400 0x55d212910000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c5520280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d212918180 con 0x55d212909680 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c5520280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d2128ba900 con 0x55d21290a400 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c5520280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55d2128ba480 con 0x55d212909b00 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212909680 0x55d212910b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212909680 0x55d212910b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51552/0 (socket says 172.21.15.28:51552) 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 -- 172.21.15.28:0/4045723312 learned_addr learned my addr 172.21.15.28:0/4045723312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c44cd640 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d21290a400 0x55d212910000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c34cb640 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212909b00 0x55d212910580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 -- 172.21.15.28:0/4045723312 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212909b00 msgr2=0x55d212910580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212909b00 0x55d212910580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 -- 172.21.15.28:0/4045723312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d21290a400 msgr2=0x55d212910000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d21290a400 0x55d212910000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 -- 172.21.15.28:0/4045723312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d20e85e780 con 0x55d212909680 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c3ccc640 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212909680 0x55d212910b00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x55d213691860 tx=0x55d212958c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a868eb865eefde0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.557+0000 7f42c14c7640 1 -- 172.21.15.28:0/4045723312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d20e84e9c0 con 0x55d212909680 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c14c7640 1 -- 172.21.15.28:0/4045723312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d20e854700 con 0x55d212909680 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c14c7640 1 -- 172.21.15.28:0/4045723312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d20e84e680 con 0x55d212909680 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c5520280 1 -- 172.21.15.28:0/4045723312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d20e85f4a0 con 0x55d212909680 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c5520280 1 -- 172.21.15.28:0/4045723312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55d20e85e000 con 0x55d212909680 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c5520280 4 waiting for MDS map... 2024-04-25T19:13:09.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c14c7640 1 -- 172.21.15.28:0/4045723312 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55d213646200 con 0x55d212909680 2024-04-25T19:13:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c14c7640 1 -- 172.21.15.28:0/4045723312 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x55d2128b3180 con 0x55d212909680 2024-04-25T19:13:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c5520280 4 Got MDS map 25 2024-04-25T19:13:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c5520280 10 main: JournalTool::main 2024-04-25T19:13:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.558+0000 7f42c5520280 4 main: JournalTool: connecting to RADOS... 2024-04-25T19:13:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.559+0000 7f42c5520280 1 Processor -- start 2024-04-25T19:13:09.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.559+0000 7f42c5520280 1 -- start start 2024-04-25T19:13:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c5520280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 0x55d212911080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c5520280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908900 0x55d212911600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c5520280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908480 0x55d212911b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c5520280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d2128ba180 con 0x55d212908480 2024-04-25T19:13:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c5520280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d212919500 con 0x55d212908900 2024-04-25T19:13:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c5520280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55d212919380 con 0x55d212908d80 2024-04-25T19:13:09.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c34cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908480 0x55d212911b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c34cb640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908480 0x55d212911b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51560/0 (socket says 172.21.15.28:51560) 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c34cb640 1 -- 172.21.15.28:0/2977442447 learned_addr learned my addr 172.21.15.28:0/2977442447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c3ccc640 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908900 0x55d212911600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c44cd640 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 0x55d212911080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c34cb640 1 -- 172.21.15.28:0/2977442447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 msgr2=0x55d212911080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c34cb640 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 0x55d212911080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c34cb640 1 -- 172.21.15.28:0/2977442447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908900 msgr2=0x55d212911600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c34cb640 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908900 0x55d212911600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.560+0000 7f42c34cb640 1 -- 172.21.15.28:0/2977442447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d20e85e3c0 con 0x55d212908480 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42c34cb640 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908480 0x55d212911b80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x55d21366b0b0 tx=0x55d213671800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9536b84a222187fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42bf4c3640 1 -- 172.21.15.28:0/2977442447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d20e84e9c0 con 0x55d212908480 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42bf4c3640 1 -- 172.21.15.28:0/2977442447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d20e854e00 con 0x55d212908480 2024-04-25T19:13:09.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42bf4c3640 1 -- 172.21.15.28:0/2977442447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d20e84e680 con 0x55d212908480 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42c5520280 1 -- 172.21.15.28:0/2977442447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908480 msgr2=0x55d212911b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42c5520280 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908480 0x55d212911b80 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x55d21366b0b0 tx=0x55d213671800 comp rx=0 tx=0).stop 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42c5520280 1 -- 172.21.15.28:0/2977442447 shutdown_connections 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42c5520280 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908480 0x55d212911b80 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42c5520280 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908900 0x55d212911600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42c5520280 1 --2- 172.21.15.28:0/2977442447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 0x55d212911080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.561+0000 7f42c5520280 1 -- 172.21.15.28:0/2977442447 >> 172.21.15.28:0/2977442447 conn(0x55d212909200 msgr2=0x55d212941800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.562+0000 7f42c5520280 1 -- 172.21.15.28:0/2977442447 shutdown_connections 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.562+0000 7f42c5520280 1 -- 172.21.15.28:0/2977442447 wait complete. 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.562+0000 7f42c5520280 1 Processor -- start 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.562+0000 7f42c5520280 1 -- start start 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.563+0000 7f42c5520280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908480 0x55d212911b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c5520280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908900 0x55d212911600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c5520280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 0x55d212911080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c5520280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d212919680 con 0x55d212908900 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c5520280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55d212918900 con 0x55d212908480 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c5520280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55d2128ba480 con 0x55d212908d80 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908900 0x55d212911600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908900 0x55d212911600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51566/0 (socket says 172.21.15.28:51566) 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 -- 172.21.15.28:0/4233533157 learned_addr learned my addr 172.21.15.28:0/4233533157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c34cb640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 0x55d212911080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c44cd640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908480 0x55d212911b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 -- 172.21.15.28:0/4233533157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 msgr2=0x55d212911080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212908d80 0x55d212911080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 -- 172.21.15.28:0/4233533157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908480 msgr2=0x55d212911b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d212908480 0x55d212911b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 -- 172.21.15.28:0/4233533157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d20e85e960 con 0x55d212908900 2024-04-25T19:13:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c3ccc640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908900 0x55d212911600 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x55d21366a210 tx=0x55d212958600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42bdcc0640 1 -- 172.21.15.28:0/4233533157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d20e84eb60 con 0x55d212908900 2024-04-25T19:13:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42bdcc0640 1 -- 172.21.15.28:0/4233533157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55d20e854fc0 con 0x55d212908900 2024-04-25T19:13:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55d20e85f680 con 0x55d212908900 2024-04-25T19:13:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42bdcc0640 1 -- 172.21.15.28:0/4233533157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55d20e84e820 con 0x55d212908900 2024-04-25T19:13:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d20e85f860 con 0x55d212908900 2024-04-25T19:13:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.564+0000 7f42c5520280 4 main: JournalTool: resolving pool 7 2024-04-25T19:13:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42bdcc0640 1 -- 172.21.15.28:0/4233533157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x55d21376c900 con 0x55d212908900 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42bdcc0640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d21290b180 0x55d212912100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42bdcc0640 1 -- 172.21.15.28:0/4233533157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55d21377b200 con 0x55d212908900 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42c5520280 4 main: JournalTool: creating IoCtx.. 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42c44cd640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d21290b180 0x55d212912100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42c5520280 4 main: Executing for rank 0 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42c5520280 1 main_event: Using alternate pool recovery_fs_metadata 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42c5520280 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55d212908000 0x55d212912680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e46) -- 0x55d213791c00 con 0x55d212908000 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.566+0000 7f42c34cb640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55d212908000 0x55d212912680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.567+0000 7f42c44cd640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d21290b180 0x55d212912100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x55d212942cf0 tx=0x55d212bd0f00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.567+0000 7f42c34cb640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55d212908000 0x55d212912680 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.567+0000 7f42c34cb640 1 -- 172.21.15.28:0/4233533157 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== osd_op_reply(1 400.00000000 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55d21297e000 con 0x55d212908000 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.567+0000 7f42c5520280 -1 Pointer 400.00000000 is absent 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.567+0000 7f42c5520280 4 main_event: consumed 0 inodes 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.567+0000 7f42c5520280 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55d21290a880 0x55d212912c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.567+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:2 7.3 7:c0ce80d4:::4.00000000:head [omap-rm-keys in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55d213790400 con 0x55d21290a880 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.567+0000 7f42c44cd640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55d21290a880 0x55d212912c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.568+0000 7f42c44cd640 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55d21290a880 0x55d212912c00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c44cd640 1 -- 172.21.15.28:0/4233533157 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== osd_op_reply(2 4.00000000 [omap-rm-keys] v46'1 uv0 ondisk = -2 ((2) No such file or directory)) ==== 154+0+0 (crc 0 0 0) 0x55d21297e500 con 0x55d21290a880 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 4 0 events 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55d212908000 msgr2=0x55d212912680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55d212908000 0x55d212912680 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55d21290a880 msgr2=0x55d212912c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55d21290a880 0x55d212912c00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d21290b180 msgr2=0x55d212912100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55d21290b180 0x55d212912100 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x55d212942cf0 tx=0x55d212bd0f00 comp rx=0 tx=0).stop 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908900 msgr2=0x55d212911600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908900 0x55d212911600 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x55d21366a210 tx=0x55d212958600 comp rx=0 tx=0).stop 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c34cb640 1 -- 172.21.15.28:0/4233533157 reap_dead start 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 shutdown_connections 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4233533157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212908900 0x55d212911600 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 >> 172.21.15.28:0/4233533157 conn(0x55d212909200 msgr2=0x55d212941800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 shutdown_connections 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4233533157 wait complete. 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4045723312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212909680 msgr2=0x55d212910b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212909680 0x55d212910b00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x55d213691860 tx=0x55d212958c60 comp rx=0 tx=0).stop 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4045723312 shutdown_connections 2024-04-25T19:13:09.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55d21290a400 0x55d212910000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55d212909b00 0x55d212910580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 --2- 172.21.15.28:0/4045723312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d212909680 0x55d212910b00 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4045723312 >> 172.21.15.28:0/4045723312 conn(0x55d212909f80 msgr2=0x55d212941000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4045723312 shutdown_connections 2024-04-25T19:13:09.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.569+0000 7f42c5520280 1 -- 172.21.15.28:0/4045723312 wait complete. 2024-04-25T19:13:09.578 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'recovery_fs:0', 'event', 'recover_dentries', 'list', '--alternate-pool', 'recovery_fs_metadata'] in time 0:00:00.077169, result: 2024-04-25T19:13:09.578 DEBUG:teuthology.orchestra.run.smithi028:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank recovery_fs:0 journal reset --force 2024-04-25T19:13:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.630+0000 7f9e54817280 1 -- start start 2024-04-25T19:13:09.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.630+0000 7f9e54817280 1 Processor -- start 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.631+0000 7f9e54817280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b7680 0x55f6474e4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.631+0000 7f9e54817280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b7b00 0x55f6474e4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.631+0000 7f9e54817280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8400 0x55f6474e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.631+0000 7f9e54817280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6467c6180 con 0x55f6467b7b00 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.631+0000 7f9e54817280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f6467c6300 con 0x55f6467b8400 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.631+0000 7f9e54817280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55f6467c6480 con 0x55f6467b7680 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b7b00 0x55f6474e4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b7b00 0x55f6474e4000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51580/0 (socket says 172.21.15.28:51580) 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 -- 172.21.15.28:0/779440190 learned_addr learned my addr 172.21.15.28:0/779440190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8400 0x55f6474e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e52439640 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b7680 0x55f6474e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 -- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b7680 msgr2=0x55f6474e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b7680 0x55f6474e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 -- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8400 msgr2=0x55f6474e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8400 0x55f6474e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 -- 172.21.15.28:0/779440190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f64270c780 con 0x55f6467b7b00 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8400 0x55f6474e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e51c38640 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b7b00 0x55f6474e4000 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x55f647537a10 tx=0x55f646808a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2128856d5225ff0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e4fc34640 1 -- 172.21.15.28:0/779440190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f6426fc680 con 0x55f6467b7b00 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e4fc34640 1 -- 172.21.15.28:0/779440190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55f642702700 con 0x55f6467b7b00 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e4fc34640 1 -- 172.21.15.28:0/779440190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f6426fcb60 con 0x55f6467b7b00 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e54817280 1 -- 172.21.15.28:0/779440190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f64270d4a0 con 0x55f6467b7b00 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e54817280 1 -- 172.21.15.28:0/779440190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55f64270c000 con 0x55f6467b7b00 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.632+0000 7f9e54817280 4 waiting for MDS map... 2024-04-25T19:13:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.633+0000 7f9e4fc34640 1 -- 172.21.15.28:0/779440190 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55f6475f2600 con 0x55f6467b7b00 2024-04-25T19:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.633+0000 7f9e4fc34640 1 -- 172.21.15.28:0/779440190 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x55f646761180 con 0x55f6467b7b00 2024-04-25T19:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.633+0000 7f9e54817280 4 Got MDS map 25 2024-04-25T19:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.633+0000 7f9e54817280 10 main: JournalTool::main 2024-04-25T19:13:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.633+0000 7f9e54817280 4 main: JournalTool: connecting to RADOS... 2024-04-25T19:13:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.634+0000 7f9e54817280 1 Processor -- start 2024-04-25T19:13:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.634+0000 7f9e54817280 1 -- start start 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e54817280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e54817280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6900 0x55f6474e5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e54817280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e54817280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6467c6600 con 0x55f6467b6900 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e54817280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f6467c6780 con 0x55f6467b6480 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e54817280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55f6467c6900 con 0x55f6467b6d80 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6900 0x55f6474e5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6900 0x55f6474e5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51586/0 (socket says 172.21.15.28:51586) 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 -- 172.21.15.28:0/2975332748 learned_addr learned my addr 172.21.15.28:0/2975332748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 -- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6d80 msgr2=0x55f6474e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 -- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6480 msgr2=0x55f6474e5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 -- 172.21.15.28:0/2975332748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f64270c3c0 con 0x55f6467b6900 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.635+0000 7f9e52439640 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6900 0x55f6474e5600 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x55f6475fcf30 tx=0x55f647601800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f1762fd746bbdea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e4dc30640 1 -- 172.21.15.28:0/2975332748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f6426fc680 con 0x55f6467b6900 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e4dc30640 1 -- 172.21.15.28:0/2975332748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55f642702fc0 con 0x55f6467b6900 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e4dc30640 1 -- 172.21.15.28:0/2975332748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f6426fcb60 con 0x55f6467b6900 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 -- 172.21.15.28:0/2975332748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6900 msgr2=0x55f6474e5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6900 0x55f6474e5600 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x55f6475fcf30 tx=0x55f647601800 comp rx=0 tx=0).stop 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 -- 172.21.15.28:0/2975332748 shutdown_connections 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6900 0x55f6474e5600 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 --2- 172.21.15.28:0/2975332748 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 -- 172.21.15.28:0/2975332748 >> 172.21.15.28:0/2975332748 conn(0x55f6467b7200 msgr2=0x55f6467ef800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 -- 172.21.15.28:0/2975332748 shutdown_connections 2024-04-25T19:13:09.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.636+0000 7f9e54817280 1 -- 172.21.15.28:0/2975332748 wait complete. 2024-04-25T19:13:09.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.637+0000 7f9e54817280 1 Processor -- start 2024-04-25T19:13:09.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.637+0000 7f9e54817280 1 -- start start 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e54817280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e54817280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6900 0x55f6474e5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e54817280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e54817280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6467c6900 con 0x55f6467b6480 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e54817280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f6467c6a80 con 0x55f6467b6d80 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e54817280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55f6467c6780 con 0x55f6467b6900 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e52c3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e52c3a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51592/0 (socket says 172.21.15.28:51592) 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.638+0000 7f9e52c3a640 1 -- 172.21.15.28:0/2911759712 learned_addr learned my addr 172.21.15.28:0/2911759712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e52439640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6900 0x55f6474e5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e52c3a640 1 -- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6900 msgr2=0x55f6474e5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6900 0x55f6474e5600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e52c3a640 1 -- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6d80 msgr2=0x55f6474e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e52c3a640 1 -- 172.21.15.28:0/2911759712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f64270c780 con 0x55f6467b6480 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x55f6475370b0 tx=0x55f646806660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e4c42d640 1 -- 172.21.15.28:0/2911759712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f6426fc9c0 con 0x55f6467b6480 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e4c42d640 1 -- 172.21.15.28:0/2911759712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55f642702e00 con 0x55f6467b6480 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e54817280 1 -- 172.21.15.28:0/2911759712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55f64270d4a0 con 0x55f6467b6480 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e4c42d640 1 -- 172.21.15.28:0/2911759712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f6426fc680 con 0x55f6467b6480 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e54817280 1 -- 172.21.15.28:0/2911759712 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f64270c960 con 0x55f6467b6480 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.639+0000 7f9e54817280 4 main: JournalTool: resolving pool 7 2024-04-25T19:13:09.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.641+0000 7f9e4c42d640 1 -- 172.21.15.28:0/2911759712 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x55f64763c900 con 0x55f6467b6480 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.641+0000 7f9e4c42d640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f6467b9180 0x55f6474e6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.641+0000 7f9e4c42d640 1 -- 172.21.15.28:0/2911759712 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55f647658c00 con 0x55f6467b6480 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.641+0000 7f9e54817280 4 main: JournalTool: creating IoCtx.. 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.641+0000 7f9e54817280 4 main: Executing for rank 0 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.641+0000 7f9e52439640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f6467b9180 0x55f6474e6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.641+0000 7f9e52439640 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f6467b9180 0x55f6474e6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55f6475fcf30 tx=0x55f6468064e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.642+0000 7f9e54817280 1 -- start start 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.642+0000 7f9e54817280 1 Processor -- start 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e54817280 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b8880 0x55f6474e6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e54817280 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8d00 0x55f6474e6c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e54817280 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b9600 0x55f6474e7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e54817280 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f6467c7b00 con 0x55f6467b9600 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e54817280 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x55f6467c7080 con 0x55f6467b8d00 2024-04-25T19:13:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e54817280 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x55f6467c6f00 con 0x55f6467b8880 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e51c38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b9600 0x55f6474e7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e51c38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b9600 0x55f6474e7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51608/0 (socket says 172.21.15.28:51608) 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e51c38640 1 -- 172.21.15.28:0/2733775239 learned_addr learned my addr 172.21.15.28:0/2733775239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e52439640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8d00 0x55f6474e6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.643+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b8880 0x55f6474e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e51c38640 1 -- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b8880 msgr2=0x55f6474e6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b8880 0x55f6474e6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e51c38640 1 -- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8d00 msgr2=0x55f6474e6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8d00 0x55f6474e6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e51c38640 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f64270d680 con 0x55f6467b9600 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b9600 0x55f6474e7180 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55f6475fc8a0 tx=0x55f646806600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43296db51d1ea0cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e49427640 1 -- 172.21.15.28:0/2733775239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f6426fc820 con 0x55f6467b9600 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e49427640 1 -- 172.21.15.28:0/2733775239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55f642702c40 con 0x55f6467b9600 2024-04-25T19:13:09.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e49427640 1 -- 172.21.15.28:0/2733775239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55f646a77040 con 0x55f6467b9600 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f64270d860 con 0x55f6467b9600 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=0}) -- 0x55f64270da40 con 0x55f6467b9600 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e54817280 4 waiting for MDS map... 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e49427640 1 -- 172.21.15.28:0/2733775239 <== mon.0 v2:172.21.15.28:3300/0 4 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x55f646b48c00 con 0x55f6467b9600 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e49427640 1 -- 172.21.15.28:0/2733775239 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 4135+0+0 (secure 0 0 0) 0x55f646761500 con 0x55f6467b9600 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e54817280 4 Got MDS map 25 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f6467b9a80 0x55f6474e7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.644+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:1 7.f 7:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f646b7c000 con 0x55f6467b9a80 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.645+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f6467b9a80 0x55f6474e7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.645+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f6467b9a80 0x55f6474e7700 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=9a7afa1a158a0384 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.646+0000 7f9e52c3a640 1 -- 172.21.15.28:0/2733775239 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v46'1 uv1 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55f64682c000 con 0x55f6467b9a80 2024-04-25T19:13:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.646+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55f646b86000 0x55f646b8e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.646+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:2 7.14 7:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f646b7c400 con 0x55f646b86000 2024-04-25T19:13:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.647+0000 7f9e52439640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55f646b86000 0x55f646b8e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.647+0000 7f9e52439640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55f646b86000 0x55f646b8e000 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=d621fe494af74ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e52439640 1 -- 172.21.15.28:0/2733775239 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v46'2 uv2 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55f64682c500 con 0x55f646b86000 2024-04-25T19:13:09.653 INFO:teuthology.orchestra.run.smithi028.stdout:writing EResetJournal entry 2024-04-25T19:13:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:3 7.14 7:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f646b7c800 con 0x55f646b86000 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- osd_op(unknown.0.0:4 7.1b 7:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f646b7cc00 con 0x55f6467b9a80 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55f646b86480 0x55f646b8e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:5 7.b 7:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f646b7d000 con 0x55f646b86480 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55f646b86900 0x55f646b8eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- osd_op(unknown.0.0:6 7.1f 7:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f646b7d400 con 0x55f646b86900 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55f646b86d80 0x55f646b8f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- osd_op(unknown.0.0:7 7.1a 7:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f646b7d800 con 0x55f646b86d80 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- osd_op(unknown.0.0:8 7.1e 7:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f646b7dc00 con 0x55f646b86480 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.648+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55f646b86480 0x55f646b8e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.649+0000 7f9e52439640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55f646b86900 0x55f646b8eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.649+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55f646b86d80 0x55f646b8f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.649+0000 7f9e52c3a640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55f646b86480 0x55f646b8e580 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=221a3d0448cf556f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.649+0000 7f9e51c38640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55f646b86d80 0x55f646b8f080 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=4656654a8ad1816 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.650+0000 7f9e52439640 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55f646b86900 0x55f646b8eb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=7c6ab1610f9df28d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.650+0000 7f9e52439640 1 -- 172.21.15.28:0/2733775239 <== osd.3 v2:172.21.15.28:6818/1928233777 2 ==== osd_op_reply(3 200.00000001 [delete] v46'3 uv2 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55f64682c500 con 0x55f646b86000 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.650+0000 7f9e52c3a640 1 -- 172.21.15.28:0/2733775239 <== osd.0 v2:172.21.15.28:6800/1137456103 2 ==== osd_op_reply(4 200.00000002 [delete] v46'1 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55f64682c000 con 0x55f6467b9a80 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.650+0000 7f9e47c24640 1 -- 172.21.15.28:0/2733775239 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- osd_op(unknown.0.0:9 7.14 7:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46) -- 0x55f647656800 con 0x55f646b86000 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.650+0000 7f9e52c3a640 1 -- 172.21.15.28:0/2733775239 <== osd.1 v2:172.21.15.28:6808/1898594658 1 ==== osd_op_reply(8 200.00000006 [delete] v46'1 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55f64682c000 con 0x55f646b86480 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e52c3a640 1 -- 172.21.15.28:0/2733775239 <== osd.1 v2:172.21.15.28:6808/1898594658 2 ==== osd_op_reply(5 200.00000003 [delete] v46'2 uv1 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55f64682c000 con 0x55f646b86480 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e52439640 1 -- 172.21.15.28:0/2733775239 <== osd.3 v2:172.21.15.28:6818/1928233777 3 ==== osd_op_reply(9 200.00000001 [write 0~48] v46'4 uv4 ondisk = 0) ==== 156+0+0 (crc 0 0 0) 0x55f64682c500 con 0x55f646b86000 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e52439640 1 -- 172.21.15.28:0/2733775239 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== osd_op_reply(6 200.00000004 [delete] v46'5 uv4 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55f64682c500 con 0x55f646b86900 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e51c38640 1 -- 172.21.15.28:0/2733775239 <== osd.4 v2:172.21.15.53:6824/2055146181 1 ==== osd_op_reply(7 200.00000005 [delete] v46'1 uv0 ondisk = -2 ((2) No such file or directory)) ==== 156+0+0 (crc 0 0 0) 0x55f64682c780 con 0x55f646b86d80 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 4 Successfully wrote new journal pointer and header for rank 3:0 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f6467b9a80 msgr2=0x55f6474e7700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x55f6467b9a80 0x55f6474e7700 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55f646b86480 msgr2=0x55f646b8e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x55f646b86480 0x55f646b8e580 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55f646b86000 msgr2=0x55f646b8e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x55f646b86000 0x55f646b8e000 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55f646b86d80 msgr2=0x55f646b8f080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x55f646b86d80 0x55f646b8f080 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e51c38640 1 -- 172.21.15.28:0/2733775239 reap_dead start 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55f646b86900 msgr2=0x55f646b8eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.651+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x55f646b86900 0x55f646b8eb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e51c38640 1 -- 172.21.15.28:0/2733775239 reap_dead start 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b9600 msgr2=0x55f6474e7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b9600 0x55f6474e7180 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55f6475fc8a0 tx=0x55f646806600 comp rx=0 tx=0).stop 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 shutdown_connections 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/2733775239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b9600 0x55f6474e7180 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 >> 172.21.15.28:0/2733775239 conn(0x55f6467b6000 msgr2=0x55f646afe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 shutdown_connections 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2733775239 wait complete. 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f6467b9180 msgr2=0x55f6474e6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f6467b9180 0x55f6474e6100 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55f6475fcf30 tx=0x55f6468064e0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6480 msgr2=0x55f6474e5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x55f6475370b0 tx=0x55f646806660 comp rx=0 tx=0).stop 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2911759712 shutdown_connections 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x55f6467b9180 0x55f6474e6100 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b6d80 0x55f6474e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b6900 0x55f6474e5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/2911759712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b6480 0x55f6474e5b80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2911759712 >> 172.21.15.28:0/2911759712 conn(0x55f6467b7200 msgr2=0x55f6467ef800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2911759712 shutdown_connections 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/2911759712 wait complete. 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/779440190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b7b00 msgr2=0x55f6474e4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b7b00 0x55f6474e4000 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x55f647537a10 tx=0x55f646808a20 comp rx=0 tx=0).stop 2024-04-25T19:13:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/779440190 shutdown_connections 2024-04-25T19:13:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55f6467b8400 0x55f6474e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f6467b7b00 0x55f6474e4000 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 --2- 172.21.15.28:0/779440190 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x55f6467b7680 0x55f6474e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.652+0000 7f9e54817280 1 -- 172.21.15.28:0/779440190 >> 172.21.15.28:0/779440190 conn(0x55f6467b7f80 msgr2=0x55f6467ef000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.653+0000 7f9e54817280 1 -- 172.21.15.28:0/779440190 shutdown_connections 2024-04-25T19:13:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.653+0000 7f9e54817280 1 -- 172.21.15.28:0/779440190 wait complete. 2024-04-25T19:13:09.662 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'recovery_fs:0', 'journal', 'reset', '--force'] in time 0:00:00.084127, result: writing EResetJournal entry 2024-04-25T19:13:09.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set recovery_fs joinable true 2024-04-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.775+0000 7fb1b5db7640 1 Processor -- start 2024-04-25T19:13:09.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.775+0000 7fb1b5db7640 1 -- start start 2024-04-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1b5db7640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb1b00b79f0 0x7fb1b00b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1b5db7640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1b00b9920 0x7fb1b00b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1b5db7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b0160720 con 0x7fb1b0160310 2024-04-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1b5db7640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb1b00b8340 con 0x7fb1b00b9920 2024-04-25T19:13:09.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1b5db7640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fb1b00d0f40 con 0x7fb1b00b79f0 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1affff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1b00b9920 0x7fb1b00b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1af7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb1b0160310 0x7fb1b00d5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41060/0 (socket says 172.21.15.28:41060) 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1affff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb1b00b9920 0x7fb1b00b9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51610/0 (socket says 172.21.15.28:51610) 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2818197305 learned_addr learned my addr 172.21.15.28:0/2818197305 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1aeffd640 1 --2- 172.21.15.28:0/2818197305 >> v2:172.21.15.53:3300/0 conn(0x7fb1b00b79f0 0x7fb1b00b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1affff640 1 -- 172.21.15.28:0/2818197305 >> v2:172.21.15.53:3300/0 conn(0x7fb1b00b79f0 msgr2=0x7fb1b00b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1affff640 1 --2- 172.21.15.28:0/2818197305 >> v2:172.21.15.53:3300/0 conn(0x7fb1b00b79f0 0x7fb1b00b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1affff640 1 -- 172.21.15.28:0/2818197305 >> v1:172.21.15.28:6789/0 conn(0x7fb1b0160310 legacy=0x7fb1b00d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/2818197305 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980605101 0 0) 0x7fb1b0160720 con 0x7fb1b0160310 2024-04-25T19:13:09.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.777+0000 7fb1affff640 1 -- 172.21.15.28:0/2818197305 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b00d1220 con 0x7fb1b00b9920 2024-04-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.778+0000 7fb1affff640 1 --2- 172.21.15.28:0/2818197305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b9920 0x7fb1b00b9d30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00d09b0 tx=0x7fb1a802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa725451c634d015 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.778+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/2818197305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8007740 con 0x7fb1b00b9920 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.778+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/2818197305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a80078e0 con 0x7fb1b00b9920 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.778+0000 7fb1ae7fc640 1 -- 172.21.15.28:0/2818197305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a8007cb0 con 0x7fb1b00b9920 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.779+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2818197305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b9920 msgr2=0x7fb1b00b9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.779+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2818197305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b9920 0x7fb1b00b9d30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb1b00d09b0 tx=0x7fb1a802a680 comp rx=0 tx=0).stop 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.779+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2818197305 shutdown_connections 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.779+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2818197305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b9920 0x7fb1b00b9d30 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.779+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2818197305 >> v2:172.21.15.53:3300/0 conn(0x7fb1b00b79f0 0x7fb1b00b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.779+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2818197305 >> 172.21.15.28:0/2818197305 conn(0x7fb1b00d2a50 msgr2=0x7fb1b00d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.779+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2818197305 shutdown_connections 2024-04-25T19:13:09.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.779+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2818197305 wait complete. 2024-04-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.781+0000 7fb1b5db7640 1 Processor -- start 2024-04-25T19:13:09.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.781+0000 7fb1b5db7640 1 -- start start 2024-04-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1b5db7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b79f0 0x7fb1b01597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1b5db7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b00b9920 0x7fb1b00c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1b5db7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1b0160310 0x7fb1b0147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1b5db7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b00d0bf0 con 0x7fb1b00b79f0 2024-04-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1b5db7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b00b9400 con 0x7fb1b00b9920 2024-04-25T19:13:09.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1b5db7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b00cc950 con 0x7fb1b0160310 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b79f0 0x7fb1b01597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1af7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b79f0 0x7fb1b01597f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51612/0 (socket says 172.21.15.28:51612) 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2405326410 learned_addr learned my addr 172.21.15.28:0/2405326410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1aeffd640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b00b9920 0x7fb1b00c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1affff640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1b0160310 0x7fb1b0147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.782+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2405326410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1b0160310 msgr2=0x7fb1b0147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb1af7fe640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1b0160310 0x7fb1b0147ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2405326410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b00b9920 msgr2=0x7fb1b00c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb1af7fe640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b00b9920 0x7fb1b00c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb1af7fe640 1 -- 172.21.15.28:0/2405326410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b0159d30 con 0x7fb1b00b79f0 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb1af7fe640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b79f0 0x7fb1b01597f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000c460 tx=0x7fb1a000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb197fff640 1 -- 172.21.15.28:0/2405326410 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00074e0 con 0x7fb1b00b79f0 2024-04-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb197fff640 1 -- 172.21.15.28:0/2405326410 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a00047e0 con 0x7fb1b00b79f0 2024-04-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b00c8a80 con 0x7fb1b00b79f0 2024-04-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb197fff640 1 -- 172.21.15.28:0/2405326410 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0004bd0 con 0x7fb1b00b79f0 2024-04-25T19:13:09.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.783+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b015a520 con 0x7fb1b00b79f0 2024-04-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.784+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b00d3000 con 0x7fb1b00b79f0 2024-04-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.785+0000 7fb197fff640 1 -- 172.21.15.28:0/2405326410 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb1a0011070 con 0x7fb1b00b79f0 2024-04-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.785+0000 7fb197fff640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb19c0744e0 0x7fb19c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.785+0000 7fb197fff640 1 -- 172.21.15.28:0/2405326410 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 6640+0+0 (secure 0 0 0) 0x7fb1a00958d0 con 0x7fb1b00b79f0 2024-04-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.785+0000 7fb1aeffd640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb19c0744e0 0x7fb19c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:09.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.786+0000 7fb1aeffd640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb19c0744e0 0x7fb19c0769f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0147980 tx=0x7fb1a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:09.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:09.789+0000 7fb197fff640 1 -- 172.21.15.28:0/2405326410 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1a005fc40 con 0x7fb1b00b79f0 2024-04-25T19:13:10.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.025+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "recovery_fs", "var": "joinable", "val": "true"} v 0) -- 0x7fb1b0076140 con 0x7fb1b00b79f0 2024-04-25T19:13:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.966+0000 7fb197fff640 1 -- 172.21.15.28:0/2405326410 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "recovery_fs", "var": "joinable", "val": "true"}]=0 recovery_fs marked joinable; MDS may join as newly active. v26) ==== 172+0+0 (secure 0 0 0) 0x7fb1a005f540 con 0x7fb1b00b79f0 2024-04-25T19:13:10.968 INFO:teuthology.orchestra.run.smithi028.stderr:recovery_fs marked joinable; MDS may join as newly active. 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb19c0744e0 msgr2=0x7fb19c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb19c0744e0 0x7fb19c0769f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0147980 tx=0x7fb1a4008040 comp rx=0 tx=0).stop 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b79f0 msgr2=0x7fb1b01597f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b79f0 0x7fb1b01597f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb1a000c460 tx=0x7fb1a000c920 comp rx=0 tx=0).stop 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 shutdown_connections 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb19c0744e0 0x7fb19c0769f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1b0160310 0x7fb1b0147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b00b9920 0x7fb1b00c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 --2- 172.21.15.28:0/2405326410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b00b79f0 0x7fb1b01597f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.970+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 >> 172.21.15.28:0/2405326410 conn(0x7fb1b00d2a50 msgr2=0x7fb1b00b9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:10.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.971+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 shutdown_connections 2024-04-25T19:13:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:10.971+0000 7fb1b5db7640 1 -- 172.21.15.28:0/2405326410 wait complete. 2024-04-25T19:13:10.974 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-25T19:13:10.971+0000 7fa80ca99640 -1 log_channel(cluster) log [ERR] : Health check update: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:13:10.981 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-04-25T19:13:10.979+0000 7f95c5b58640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-04-25T19:13:10.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.108+0000 7f98daad8640 1 Processor -- start 2024-04-25T19:13:11.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.108+0000 7f98daad8640 1 -- start start 2024-04-25T19:13:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98daad8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98d40b7fd0 0x7f98d4160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98daad8640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f98d4164800 0x7f98d40b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98daad8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98d40d15b0 con 0x7f98d40b4c80 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98daad8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f98d4164c10 con 0x7f98d40b7fd0 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98daad8640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f98d40d12d0 con 0x7f98d4164800 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98d3fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98d40b7fd0 0x7f98d4160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98d8874640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f98d40b4c80 0x7f98d40b78c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36928/0 (socket says 172.21.15.28:36928) 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98d8874640 1 -- 172.21.15.28:0/2116226722 learned_addr learned my addr 172.21.15.28:0/2116226722 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.109+0000 7f98d9075640 1 --2- 172.21.15.28:0/2116226722 >> v2:172.21.15.53:3300/0 conn(0x7f98d4164800 0x7f98d40b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.110+0000 7f98d3fff640 1 -- 172.21.15.28:0/2116226722 >> v2:172.21.15.53:3300/0 conn(0x7f98d4164800 msgr2=0x7f98d40b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.110+0000 7f98d3fff640 1 --2- 172.21.15.28:0/2116226722 >> v2:172.21.15.53:3300/0 conn(0x7f98d4164800 0x7f98d40b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.110+0000 7f98d3fff640 1 -- 172.21.15.28:0/2116226722 >> v1:172.21.15.28:6789/0 conn(0x7f98d40b4c80 legacy=0x7f98d40b78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.110+0000 7f98d3fff640 1 -- 172.21.15.28:0/2116226722 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d4163d20 con 0x7f98d40b7fd0 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.110+0000 7f98d3fff640 1 --2- 172.21.15.28:0/2116226722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b7fd0 0x7f98d4160220 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f98d40d56e0 tx=0x7f98bc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9b4a52c5edbe14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.110+0000 7f98d37fe640 1 -- 172.21.15.28:0/2116226722 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc037070 con 0x7f98d40b7fd0 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.110+0000 7f98d37fe640 1 -- 172.21.15.28:0/2116226722 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98bc032440 con 0x7f98d40b7fd0 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98d37fe640 1 -- 172.21.15.28:0/2116226722 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98bc032810 con 0x7f98d40b7fd0 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98daad8640 1 -- 172.21.15.28:0/2116226722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b7fd0 msgr2=0x7f98d4160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98daad8640 1 --2- 172.21.15.28:0/2116226722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b7fd0 0x7f98d4160220 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f98d40d56e0 tx=0x7f98bc02a1a0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98daad8640 1 -- 172.21.15.28:0/2116226722 shutdown_connections 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98daad8640 1 --2- 172.21.15.28:0/2116226722 >> v2:172.21.15.53:3300/0 conn(0x7f98d4164800 0x7f98d40b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98daad8640 1 --2- 172.21.15.28:0/2116226722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b7fd0 0x7f98d4160220 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98daad8640 1 -- 172.21.15.28:0/2116226722 >> 172.21.15.28:0/2116226722 conn(0x7f98d40d2980 msgr2=0x7f98d40d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98daad8640 1 -- 172.21.15.28:0/2116226722 shutdown_connections 2024-04-25T19:13:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.111+0000 7f98daad8640 1 -- 172.21.15.28:0/2116226722 wait complete. 2024-04-25T19:13:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.113+0000 7f98daad8640 1 Processor -- start 2024-04-25T19:13:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.113+0000 7f98daad8640 1 -- start start 2024-04-25T19:13:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.114+0000 7f98daad8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b4c80 0x7f98d41510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.114+0000 7f98daad8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d4164800 0x7f98d4159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.114+0000 7f98daad8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d415df50 0x7f98d415e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.114+0000 7f98daad8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98d40b5240 con 0x7f98d40b4c80 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.114+0000 7f98daad8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f98d41642e0 con 0x7f98d4164800 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.114+0000 7f98daad8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f98d40b53c0 con 0x7f98d415df50 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b4c80 0x7f98d41510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b4c80 0x7f98d41510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59380/0 (socket says 172.21.15.28:59380) 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 -- 172.21.15.28:0/3823528447 learned_addr learned my addr 172.21.15.28:0/3823528447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d3fff640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d4164800 0x7f98d4159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 -- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d415df50 msgr2=0x7f98d415e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d9075640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d415df50 0x7f98d415e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d415df50 0x7f98d415e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 -- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d4164800 msgr2=0x7f98d4159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d4164800 0x7f98d4159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 -- 172.21.15.28:0/3823528447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d415e8a0 con 0x7f98d40b4c80 2024-04-25T19:13:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d3fff640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d4164800 0x7f98d4159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d8874640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b4c80 0x7f98d41510a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00ca70 tx=0x7f98cc00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d17fa640 1 -- 172.21.15.28:0/3823528447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc017070 con 0x7f98d40b4c80 2024-04-25T19:13:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d17fa640 1 -- 172.21.15.28:0/3823528447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98cc00ab70 con 0x7f98d40b4c80 2024-04-25T19:13:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98d41515e0 con 0x7f98d40b4c80 2024-04-25T19:13:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98d17fa640 1 -- 172.21.15.28:0/3823528447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98cc012400 con 0x7f98d40b4c80 2024-04-25T19:13:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.115+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98d4160b00 con 0x7f98d40b4c80 2024-04-25T19:13:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.116+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98d4050740 con 0x7f98d40b4c80 2024-04-25T19:13:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.117+0000 7f98d17fa640 1 -- 172.21.15.28:0/3823528447 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f98cc01a070 con 0x7f98d40b4c80 2024-04-25T19:13:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.118+0000 7f98d17fa640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98b40744e0 0x7f98b40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.118+0000 7f98d17fa640 1 -- 172.21.15.28:0/3823528447 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f98cc095330 con 0x7f98d40b4c80 2024-04-25T19:13:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.118+0000 7f98d3fff640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98b40744e0 0x7f98b40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.118+0000 7f98d3fff640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98b40744e0 0x7f98b40769f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f98bc034000 tx=0x7f98bc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:11.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.122+0000 7f98d17fa640 1 -- 172.21.15.28:0/3823528447 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f98d415e8a0 con 0x7f98d40b4c80 2024-04-25T19:13:11.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.346+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98d40bb110 con 0x7f98d40b4c80 2024-04-25T19:13:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.347+0000 7f98d17fa640 1 -- 172.21.15.28:0/3823528447 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+6476 (secure 0 0 0) 0x7f98cc05f680 con 0x7f98d40b4c80 2024-04-25T19:13:11.349 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:11.349 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:10.972231+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:replay","state_seq":1,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T19:13:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-04-25T19:13:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98b40744e0 msgr2=0x7f98b40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98b40744e0 0x7f98b40769f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f98bc034000 tx=0x7f98bc02e000 comp rx=0 tx=0).stop 2024-04-25T19:13:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b4c80 msgr2=0x7f98d41510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b4c80 0x7f98d41510a0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00ca70 tx=0x7f98cc00cf30 comp rx=0 tx=0).stop 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 shutdown_connections 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f98b40744e0 0x7f98b40769f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f98d415df50 0x7f98d415e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98d4164800 0x7f98d4159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 --2- 172.21.15.28:0/3823528447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d40b4c80 0x7f98d41510a0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 >> 172.21.15.28:0/3823528447 conn(0x7f98d40d2980 msgr2=0x7f98d40b5540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 shutdown_connections 2024-04-25T19:13:11.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:11.349+0000 7f98daad8640 1 -- 172.21.15.28:0/3823528447 wait complete. 2024-04-25T19:13:11.765 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T19:13:03.631282+0000', 'modified': '2024-04-25T19:13:10.972231+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5589}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5589': {'gid': 5589, 'name': 'c', 'rank': 0, 'incarnation': 27, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.28:6837/1842070529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6836', 'nonce': 1842070529}, {'type': 'v1', 'addr': '172.21.15.28:6837', 'nonce': 1842070529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T19:13:11.765 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5589:up:replay 2024-04-25T19:13:12.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.891+0000 7fafaf851640 1 Processor -- start 2024-04-25T19:13:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.892+0000 7fafaf851640 1 -- start start 2024-04-25T19:13:12.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafaf851640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fafa80b4e30 0x7fafa80bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:12.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafaf851640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 0x7fafa80b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:12.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafaf851640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fafa80b7650 0x7fafa80b7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:12.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafaf851640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fafa80d0b50 con 0x7fafa80b4e30 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafaf851640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fafa808e3a0 con 0x7fafa80b7650 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafaf851640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fafa80bbb90 con 0x7fafa80b9470 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafad5ed640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fafa80b4e30 0x7fafa80bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafad5ed640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fafa80b4e30 0x7fafa80bb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59390/0 (socket says 172.21.15.28:59390) 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafad5ed640 1 -- 172.21.15.28:0/1322415670 learned_addr learned my addr 172.21.15.28:0/1322415670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafacdec640 1 --2- 172.21.15.28:0/1322415670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 0x7fafa80b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.893+0000 7fafaddee640 1 --2- 172.21.15.28:0/1322415670 >> v2:172.21.15.53:3300/0 conn(0x7fafa80b7650 0x7fafa80b7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.894+0000 7fafad5ed640 1 -- 172.21.15.28:0/1322415670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 msgr2=0x7fafa80b9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.894+0000 7fafad5ed640 1 --2- 172.21.15.28:0/1322415670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 0x7fafa80b9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.894+0000 7fafad5ed640 1 -- 172.21.15.28:0/1322415670 >> v2:172.21.15.53:3300/0 conn(0x7fafa80b7650 msgr2=0x7fafa80b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.894+0000 7fafad5ed640 1 --2- 172.21.15.28:0/1322415670 >> v2:172.21.15.53:3300/0 conn(0x7fafa80b7650 0x7fafa80b7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.894+0000 7fafad5ed640 1 -- 172.21.15.28:0/1322415670 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa80b7fa0 con 0x7fafa80b4e30 2024-04-25T19:13:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.894+0000 7fafad5ed640 1 --2- 172.21.15.28:0/1322415670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b4e30 0x7fafa80bb650 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fafa8075750 tx=0x7fafa002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb3738c960254e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.894+0000 7faf9ffff640 1 -- 172.21.15.28:0/1322415670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa0037070 con 0x7fafa80b4e30 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7faf9ffff640 1 -- 172.21.15.28:0/1322415670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafa0032440 con 0x7fafa80b4e30 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7faf9ffff640 1 -- 172.21.15.28:0/1322415670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa0032810 con 0x7fafa80b4e30 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 -- 172.21.15.28:0/1322415670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b4e30 msgr2=0x7fafa80bb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 --2- 172.21.15.28:0/1322415670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b4e30 0x7fafa80bb650 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fafa8075750 tx=0x7fafa002a140 comp rx=0 tx=0).stop 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 -- 172.21.15.28:0/1322415670 shutdown_connections 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 --2- 172.21.15.28:0/1322415670 >> v2:172.21.15.53:3300/0 conn(0x7fafa80b7650 0x7fafa80b7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 --2- 172.21.15.28:0/1322415670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 0x7fafa80b9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 --2- 172.21.15.28:0/1322415670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b4e30 0x7fafa80bb650 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 -- 172.21.15.28:0/1322415670 >> 172.21.15.28:0/1322415670 conn(0x7fafa80d2650 msgr2=0x7fafa80d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 -- 172.21.15.28:0/1322415670 shutdown_connections 2024-04-25T19:13:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.895+0000 7fafaf851640 1 -- 172.21.15.28:0/1322415670 wait complete. 2024-04-25T19:13:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.897+0000 7fafaf851640 1 Processor -- start 2024-04-25T19:13:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.897+0000 7fafaf851640 1 -- start start 2024-04-25T19:13:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafaf851640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80b4e30 0x7fafa8158550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafaf851640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b7650 0x7fafa8153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafaf851640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 0x7fafa8161fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafaf851640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fafa80d0500 con 0x7fafa80b7650 2024-04-25T19:13:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafaf851640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fafa80d0220 con 0x7fafa80b4e30 2024-04-25T19:13:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafaf851640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fafa80d07e0 con 0x7fafa80b9470 2024-04-25T19:13:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b7650 0x7fafa8153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b7650 0x7fafa8153470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59400/0 (socket says 172.21.15.28:59400) 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 -- 172.21.15.28:0/154685200 learned_addr learned my addr 172.21.15.28:0/154685200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafad5ed640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80b4e30 0x7fafa8158550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafaddee640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 0x7fafa8161fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 -- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 msgr2=0x7fafa8161fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 0x7fafa8161fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 -- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80b4e30 msgr2=0x7fafa8158550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80b4e30 0x7fafa8158550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 -- 172.21.15.28:0/154685200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafa81ea450 con 0x7fafa80b7650 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafad5ed640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80b4e30 0x7fafa8158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7fafacdec640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b7650 0x7fafa8153470 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7faf9800d5a0 tx=0x7faf9800da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.899+0000 7faf9dffb640 1 -- 172.21.15.28:0/154685200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98019070 con 0x7fafa80b7650 2024-04-25T19:13:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.900+0000 7faf9dffb640 1 -- 172.21.15.28:0/154685200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf98004240 con 0x7fafa80b7650 2024-04-25T19:13:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.900+0000 7faf9dffb640 1 -- 172.21.15.28:0/154685200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98003610 con 0x7fafa80b7650 2024-04-25T19:13:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.900+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafa81ea730 con 0x7fafa80b7650 2024-04-25T19:13:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.900+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafa81eb950 con 0x7fafa80b7650 2024-04-25T19:13:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.901+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafa81ea1a0 con 0x7fafa80b7650 2024-04-25T19:13:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.901+0000 7faf9dffb640 1 -- 172.21.15.28:0/154685200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7faf9801c070 con 0x7fafa80b7650 2024-04-25T19:13:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.902+0000 7faf9dffb640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7faf8c0744e0 0x7faf8c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.902+0000 7faf9dffb640 1 -- 172.21.15.28:0/154685200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7faf98094e00 con 0x7fafa80b7650 2024-04-25T19:13:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.902+0000 7fafad5ed640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7faf8c0744e0 0x7faf8c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.902+0000 7fafad5ed640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7faf8c0744e0 0x7faf8c0769f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fafa0034000 tx=0x7fafa002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:12.906+0000 7faf9dffb640 1 -- 172.21.15.28:0/154685200 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf9805eb80 con 0x7fafa80b7650 2024-04-25T19:13:13.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.134+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fafa81eac50 con 0x7fafa80b7650 2024-04-25T19:13:13.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.135+0000 7faf9dffb640 1 -- 172.21.15.28:0/154685200 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+6476 (secure 0 0 0) 0x7faf98062e80 con 0x7fafa80b7650 2024-04-25T19:13:13.137 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:13.137 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:11.976690+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:rejoin","state_seq":5,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":3}]} 2024-04-25T19:13:13.137 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-04-25T19:13:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7faf8c0744e0 msgr2=0x7faf8c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7faf8c0744e0 0x7faf8c0769f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fafa0034000 tx=0x7fafa002e000 comp rx=0 tx=0).stop 2024-04-25T19:13:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b7650 msgr2=0x7fafa8153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b7650 0x7fafa8153470 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7faf9800d5a0 tx=0x7faf9800da60 comp rx=0 tx=0).stop 2024-04-25T19:13:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 shutdown_connections 2024-04-25T19:13:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7faf8c0744e0 0x7faf8c0769f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fafa80b9470 0x7fafa8161fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafa80b7650 0x7fafa8153470 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 --2- 172.21.15.28:0/154685200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fafa80b4e30 0x7fafa8158550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 >> 172.21.15.28:0/154685200 conn(0x7fafa80d2650 msgr2=0x7fafa80b5560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 shutdown_connections 2024-04-25T19:13:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:13.137+0000 7fafaf851640 1 -- 172.21.15.28:0/154685200 wait complete. 2024-04-25T19:13:13.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T19:13:03.631282+0000', 'modified': '2024-04-25T19:13:11.976690+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5589}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5589': {'gid': 5589, 'name': 'c', 'rank': 0, 'incarnation': 27, 'state': 'up:rejoin', 'state_seq': 5, 'addr': '172.21.15.28:6837/1842070529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6836', 'nonce': 1842070529}, {'type': 'v1', 'addr': '172.21.15.28:6837', 'nonce': 1842070529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T19:13:13.552 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5589:up:rejoin 2024-04-25T19:13:14.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:14.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.678+0000 7fbe8e24c640 1 Processor -- start 2024-04-25T19:13:14.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.678+0000 7fbe8e24c640 1 -- start start 2024-04-25T19:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.679+0000 7fbe8e24c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbe8806e5e0 0x7fbe88161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe8e24c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbe8815f7d0 0x7fbe8815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe8e24c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe88163e80 con 0x7fbe880b5130 2024-04-25T19:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe8e24c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fbe88163c30 con 0x7fbe8815f7d0 2024-04-25T19:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe8e24c640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fbe88164a90 con 0x7fbe8806e5e0 2024-04-25T19:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe87fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbe8815f7d0 0x7fbe8815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe877fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbe880b5130 0x7fbe8806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36940/0 (socket says 172.21.15.28:36940) 2024-04-25T19:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe877fe640 1 -- 172.21.15.28:0/1205393182 learned_addr learned my addr 172.21.15.28:0/1205393182 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe867fc640 1 -- 172.21.15.28:0/1205393182 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2834478474 0 0) 0x7fbe88163e80 con 0x7fbe880b5130 2024-04-25T19:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe867fc640 1 -- 172.21.15.28:0/1205393182 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe70003520 con 0x7fbe880b5130 2024-04-25T19:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe86ffd640 1 --2- 172.21.15.28:0/1205393182 >> v2:172.21.15.53:3300/0 conn(0x7fbe8806e5e0 0x7fbe88161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe87fff640 1 -- 172.21.15.28:0/1205393182 >> v2:172.21.15.53:3300/0 conn(0x7fbe8806e5e0 msgr2=0x7fbe88161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe87fff640 1 --2- 172.21.15.28:0/1205393182 >> v2:172.21.15.53:3300/0 conn(0x7fbe8806e5e0 0x7fbe88161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe87fff640 1 -- 172.21.15.28:0/1205393182 >> v1:172.21.15.28:6789/0 conn(0x7fbe880b5130 legacy=0x7fbe8806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.680+0000 7fbe87fff640 1 -- 172.21.15.28:0/1205393182 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe88164160 con 0x7fbe8815f7d0 2024-04-25T19:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.681+0000 7fbe87fff640 1 --2- 172.21.15.28:0/1205393182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe8815f7d0 0x7fbe8815fbe0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbe8808e180 tx=0x7fbe8002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f7d418752bf0246 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.681+0000 7fbe867fc640 1 -- 172.21.15.28:0/1205393182 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe80007740 con 0x7fbe8815f7d0 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.681+0000 7fbe867fc640 1 -- 172.21.15.28:0/1205393182 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe800078e0 con 0x7fbe8815f7d0 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.681+0000 7fbe867fc640 1 -- 172.21.15.28:0/1205393182 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe80007cb0 con 0x7fbe8815f7d0 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.682+0000 7fbe8e24c640 1 -- 172.21.15.28:0/1205393182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe8815f7d0 msgr2=0x7fbe8815fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.682+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/1205393182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe8815f7d0 0x7fbe8815fbe0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fbe8808e180 tx=0x7fbe8002a680 comp rx=0 tx=0).stop 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.682+0000 7fbe8e24c640 1 -- 172.21.15.28:0/1205393182 shutdown_connections 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.682+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/1205393182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe8815f7d0 0x7fbe8815fbe0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.682+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/1205393182 >> v2:172.21.15.53:3300/0 conn(0x7fbe8806e5e0 0x7fbe88161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.682+0000 7fbe8e24c640 1 -- 172.21.15.28:0/1205393182 >> 172.21.15.28:0/1205393182 conn(0x7fbe880b7ad0 msgr2=0x7fbe880b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.682+0000 7fbe8e24c640 1 -- 172.21.15.28:0/1205393182 shutdown_connections 2024-04-25T19:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.682+0000 7fbe8e24c640 1 -- 172.21.15.28:0/1205393182 wait complete. 2024-04-25T19:13:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.684+0000 7fbe8e24c640 1 Processor -- start 2024-04-25T19:13:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.684+0000 7fbe8e24c640 1 -- start start 2024-04-25T19:13:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe8e24c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe880b5130 0x7fbe880c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe8e24c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe8815f7d0 0x7fbe88156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe8e24c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe881572b0 0x7fbe88154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe8e24c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbe88160120 con 0x7fbe880b5130 2024-04-25T19:13:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe8e24c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbe88164720 con 0x7fbe881572b0 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe8e24c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbe881602a0 con 0x7fbe8815f7d0 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe880b5130 0x7fbe880c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe877fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe880b5130 0x7fbe880c8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59422/0 (socket says 172.21.15.28:59422) 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.685+0000 7fbe877fe640 1 -- 172.21.15.28:0/2984981573 learned_addr learned my addr 172.21.15.28:0/2984981573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe877fe640 1 -- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe8815f7d0 msgr2=0x7fbe88156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe87fff640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe881572b0 0x7fbe88154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe86ffd640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe8815f7d0 0x7fbe88156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe877fe640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe8815f7d0 0x7fbe88156d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe877fe640 1 -- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe881572b0 msgr2=0x7fbe88154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe877fe640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe881572b0 0x7fbe88154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe877fe640 1 -- 172.21.15.28:0/2984981573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe881579a0 con 0x7fbe880b5130 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe87fff640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe881572b0 0x7fbe88154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe86ffd640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe8815f7d0 0x7fbe88156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe877fe640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe880b5130 0x7fbe880c8940 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800c390 tx=0x7fbe7800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe6ffff640 1 -- 172.21.15.28:0/2984981573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78017070 con 0x7fbe880b5130 2024-04-25T19:13:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe6ffff640 1 -- 172.21.15.28:0/2984981573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe78005d00 con 0x7fbe880b5130 2024-04-25T19:13:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe6ffff640 1 -- 172.21.15.28:0/2984981573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe78007310 con 0x7fbe880b5130 2024-04-25T19:13:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe88157c80 con 0x7fbe880b5130 2024-04-25T19:13:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.686+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe8815a890 con 0x7fbe880b5130 2024-04-25T19:13:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.687+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe881581a0 con 0x7fbe880b5130 2024-04-25T19:13:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.688+0000 7fbe6ffff640 1 -- 172.21.15.28:0/2984981573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbe7801a070 con 0x7fbe880b5130 2024-04-25T19:13:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.688+0000 7fbe6ffff640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe740744e0 0x7fbe740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.688+0000 7fbe6ffff640 1 -- 172.21.15.28:0/2984981573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fbe78094900 con 0x7fbe880b5130 2024-04-25T19:13:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.688+0000 7fbe86ffd640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe740744e0 0x7fbe740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.689+0000 7fbe86ffd640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe740744e0 0x7fbe740769f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.693+0000 7fbe6ffff640 1 -- 172.21.15.28:0/2984981573 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe881581a0 con 0x7fbe880b5130 2024-04-25T19:13:14.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.917+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe880d4810 con 0x7fbe880b5130 2024-04-25T19:13:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.918+0000 7fbe6ffff640 1 -- 172.21.15.28:0/2984981573 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6483 (secure 0 0 0) 0x7fbe880d4810 con 0x7fbe880b5130 2024-04-25T19:13:14.920 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:14.920 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:14.920 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-04-25T19:13:14.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe740744e0 msgr2=0x7fbe740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:14.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe740744e0 0x7fbe740769f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c006fd0 tx=0x7fbe7c008040 comp rx=0 tx=0).stop 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe880b5130 msgr2=0x7fbe880c8940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe880b5130 0x7fbe880c8940 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800c390 tx=0x7fbe7800c850 comp rx=0 tx=0).stop 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 shutdown_connections 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbe740744e0 0x7fbe740769f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbe881572b0 0x7fbe88154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbe8815f7d0 0x7fbe88156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 --2- 172.21.15.28:0/2984981573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe880b5130 0x7fbe880c8940 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 >> 172.21.15.28:0/2984981573 conn(0x7fbe880b7ad0 msgr2=0x7fbe8806ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 shutdown_connections 2024-04-25T19:13:14.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:14.920+0000 7fbe8e24c640 1 -- 172.21.15.28:0/2984981573 wait complete. 2024-04-25T19:13:15.336 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T19:13:03.631282+0000', 'modified': '2024-04-25T19:13:13.979864+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5589}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5589': {'gid': 5589, 'name': 'c', 'rank': 0, 'incarnation': 27, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.28:6837/1842070529', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6836', 'nonce': 1842070529}, {'type': 'v1', 'addr': '172.21.15.28:6837', 'nonce': 1842070529}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'recovery_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5589, 'qdb_cluster': [5589]} 2024-04-25T19:13:15.337 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T19:13:15.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T19:13:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.452+0000 7fe79ac96640 1 Processor -- start 2024-04-25T19:13:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.452+0000 7fe79ac96640 1 -- start start 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.453+0000 7fe79ac96640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe794160310 0x7fe7940d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.453+0000 7fe79ac96640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7940b79f0 0x7fe7940b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.453+0000 7fe79ac96640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe794160720 con 0x7fe79408cce0 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.453+0000 7fe79ac96640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fe7940b8340 con 0x7fe794160310 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.453+0000 7fe79ac96640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fe7940d5640 con 0x7fe7940b79f0 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.453+0000 7fe798a32640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe794160310 0x7fe7940d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.454+0000 7fe798a32640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe794160310 0x7fe7940d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59426/0 (socket says 172.21.15.28:59426) 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.454+0000 7fe798a32640 1 -- 172.21.15.28:0/1189280726 learned_addr learned my addr 172.21.15.28:0/1189280726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.454+0000 7fe793fff640 1 --2- 172.21.15.28:0/1189280726 >> v2:172.21.15.53:3300/0 conn(0x7fe7940b79f0 0x7fe7940b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.454+0000 7fe798a32640 1 -- 172.21.15.28:0/1189280726 >> v2:172.21.15.53:3300/0 conn(0x7fe7940b79f0 msgr2=0x7fe7940b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.454+0000 7fe798a32640 1 --2- 172.21.15.28:0/1189280726 >> v2:172.21.15.53:3300/0 conn(0x7fe7940b79f0 0x7fe7940b7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.454+0000 7fe798a32640 1 -- 172.21.15.28:0/1189280726 >> v1:172.21.15.53:6789/0 conn(0x7fe79408cce0 legacy=0x7fe79408d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.454+0000 7fe798a32640 1 -- 172.21.15.28:0/1189280726 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7940d0f40 con 0x7fe794160310 2024-04-25T19:13:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe798a32640 1 --2- 172.21.15.28:0/1189280726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe794160310 0x7fe7940d5100 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe7940d2600 tx=0x7fe78802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b3b5ef9a0c80b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe7937fe640 1 -- 172.21.15.28:0/1189280726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788037070 con 0x7fe794160310 2024-04-25T19:13:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe7937fe640 1 -- 172.21.15.28:0/1189280726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe788032440 con 0x7fe794160310 2024-04-25T19:13:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe7937fe640 1 -- 172.21.15.28:0/1189280726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788032810 con 0x7fe794160310 2024-04-25T19:13:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe79ac96640 1 -- 172.21.15.28:0/1189280726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe794160310 msgr2=0x7fe7940d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe79ac96640 1 --2- 172.21.15.28:0/1189280726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe794160310 0x7fe7940d5100 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fe7940d2600 tx=0x7fe78802a1a0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe79ac96640 1 -- 172.21.15.28:0/1189280726 shutdown_connections 2024-04-25T19:13:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe79ac96640 1 --2- 172.21.15.28:0/1189280726 >> v2:172.21.15.53:3300/0 conn(0x7fe7940b79f0 0x7fe7940b7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe79ac96640 1 --2- 172.21.15.28:0/1189280726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe794160310 0x7fe7940d5100 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe79ac96640 1 -- 172.21.15.28:0/1189280726 >> 172.21.15.28:0/1189280726 conn(0x7fe7940d2a50 msgr2=0x7fe7940d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe79ac96640 1 -- 172.21.15.28:0/1189280726 shutdown_connections 2024-04-25T19:13:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.455+0000 7fe79ac96640 1 -- 172.21.15.28:0/1189280726 wait complete. 2024-04-25T19:13:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.457+0000 7fe79ac96640 1 Processor -- start 2024-04-25T19:13:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.457+0000 7fe79ac96640 1 -- start start 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe79ac96640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe79408cce0 0x7fe7941f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe79ac96640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7940b79f0 0x7fe7941f9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe79ac96640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe794160310 0x7fe7940cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe79ac96640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7940b9400 con 0x7fe79408cce0 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe79ac96640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe7940d1500 con 0x7fe7940b79f0 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe79ac96640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe7940d0bf0 con 0x7fe794160310 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe79408cce0 0x7fe7941f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe79408cce0 0x7fe7941f94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59440/0 (socket says 172.21.15.28:59440) 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 -- 172.21.15.28:0/3995040662 learned_addr learned my addr 172.21.15.28:0/3995040662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe793fff640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7940b79f0 0x7fe7941f9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe799233640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe794160310 0x7fe7940cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 -- 172.21.15.28:0/3995040662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe794160310 msgr2=0x7fe7940cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe794160310 0x7fe7940cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 -- 172.21.15.28:0/3995040662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7940b79f0 msgr2=0x7fe7941f9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7940b79f0 0x7fe7941f9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 -- 172.21.15.28:0/3995040662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7941facd0 con 0x7fe79408cce0 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe798a32640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe79408cce0 0x7fe7941f94a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe788000c00 tx=0x7fe7880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.459+0000 7fe7917fa640 1 -- 172.21.15.28:0/3995040662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788041070 con 0x7fe79408cce0 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.460+0000 7fe7917fa640 1 -- 172.21.15.28:0/3995040662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78803c870 con 0x7fe79408cce0 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.460+0000 7fe79ac96640 1 -- 172.21.15.28:0/3995040662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7941fa9f0 con 0x7fe79408cce0 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.460+0000 7fe7917fa640 1 -- 172.21.15.28:0/3995040662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788037040 con 0x7fe79408cce0 2024-04-25T19:13:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.460+0000 7fe79ac96640 1 -- 172.21.15.28:0/3995040662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7940c9af0 con 0x7fe79408cce0 2024-04-25T19:13:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.462+0000 7fe7917fa640 1 -- 172.21.15.28:0/3995040662 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe788044070 con 0x7fe79408cce0 2024-04-25T19:13:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.462+0000 7fe7917fa640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe7780744e0 0x7fe7780769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.462+0000 7fe7917fa640 1 -- 172.21.15.28:0/3995040662 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fe7880b5ee0 con 0x7fe79408cce0 2024-04-25T19:13:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.462+0000 7fe793fff640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe7780744e0 0x7fe7780769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.463+0000 7fe793fff640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe7780744e0 0x7fe7780769f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe7940d3520 tx=0x7fe780008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.463+0000 7fe7a1c23740 1 Processor -- start 2024-04-25T19:13:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.463+0000 7fe7a1c23740 1 -- start start 2024-04-25T19:13:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.464+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e007d270 0x5574e0449600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.464+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e0445fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.464+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e041f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.464+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5574e0383740 con 0x5574e007d270 2024-04-25T19:13:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.464+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5574e031f3a0 con 0x5574e00543b0 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.464+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x5574e037f700 con 0x5574e005c590 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e007d270 0x5574e0449600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e007d270 0x5574e0449600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59444/0 (socket says 172.21.15.28:59444) 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 -- 172.21.15.28:0/3600438419 learned_addr learned my addr 172.21.15.28:0/3600438419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe793fff640 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e0445fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 -- 172.21.15.28:0/3600438419 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 msgr2=0x5574e041f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe799233640 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e041f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e041f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 -- 172.21.15.28:0/3600438419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 msgr2=0x5574e0445fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e0445fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 -- 172.21.15.28:0/3600438419 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e033b750 con 0x5574e007d270 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe798a32640 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e007d270 0x5574e0449600 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe78807bd70 tx=0x7fe78807f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19075a56ff5afd79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe77e7fc640 1 -- 172.21.15.28:0/3600438419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78804b040 con 0x5574e007d270 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe77e7fc640 1 -- 172.21.15.28:0/3600438419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe788086040 con 0x5574e007d270 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.465+0000 7fe77e7fc640 1 -- 172.21.15.28:0/3600438419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788088360 con 0x5574e007d270 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 -- 172.21.15.28:0/3600438419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e007d270 msgr2=0x5574e0449600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e007d270 0x5574e0449600 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fe78807bd70 tx=0x7fe78807f640 comp rx=0 tx=0).stop 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 -- 172.21.15.28:0/3600438419 shutdown_connections 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e041f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e0445fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/3600438419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e007d270 0x5574e0449600 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 -- 172.21.15.28:0/3600438419 >> 172.21.15.28:0/3600438419 conn(0x5574e00713c0 msgr2=0x5574e041e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 -- 172.21.15.28:0/3600438419 shutdown_connections 2024-04-25T19:13:15.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.466+0000 7fe7a1c23740 1 -- 172.21.15.28:0/3600438419 wait complete. 2024-04-25T19:13:15.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.467+0000 7fe7a1c23740 1 -- start start 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.467+0000 7fe7a1c23740 1 Processor -- start 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.468+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e04432f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.468+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e0443830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e007d270 0x5574e044b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5574e0383740 con 0x5574e00543b0 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5574e030afe0 con 0x5574e007d270 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x5574e0463490 con 0x5574e005c590 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7a1c23740 10 client.0 _ll_register_callbacks cb 0x5574e0321150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e04432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e04432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59452/0 (socket says 172.21.15.28:59452) 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 -- 172.21.15.28:0/783759569 learned_addr learned my addr 172.21.15.28:0/783759569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe799233640 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e007d270 0x5574e044b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe793fff640 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e0443830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 -- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 msgr2=0x5574e0443830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e0443830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 -- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e007d270 msgr2=0x5574e044b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e007d270 0x5574e044b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 -- 172.21.15.28:0/783759569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574e00952d0 con 0x5574e00543b0 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe793fff640 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e0443830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe798a32640 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e04432f0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe78802a1e0 tx=0x7fe788007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dbbf01ec433a61d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7737fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7737fe640 1 -- 172.21.15.28:0/783759569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78804b070 con 0x5574e00543b0 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7737fe640 1 -- 172.21.15.28:0/783759569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fe788088780 con 0x5574e00543b0 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7a1c23740 1 -- 172.21.15.28:0/783759569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe788086040 con 0x5574e00543b0 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7737fe640 1 -- 172.21.15.28:0/783759569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe788088360 con 0x5574e00543b0 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7a1c23740 10 client.5865 fetch_fsmap learned FSMap version 30 2024-04-25T19:13:15.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.469+0000 7fe7a1c23740 1 -- 172.21.15.28:0/783759569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5574e015f6b0 con 0x5574e00543b0 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7737fe640 1 -- 172.21.15.28:0/783759569 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe788086040 con 0x5574e00543b0 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7737fe640 1 -- 172.21.15.28:0/783759569 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 4139+0+0 (secure 0 0 0) 0x7fe7880a10d0 con 0x5574e00543b0 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7a1c23740 10 client.5865 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7a1c23740 10 client.5865 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7a1c23740 20 client.5865 populate_metadata read hostname 'smithi028' 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5574e003be40 0x5574e0480c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7a1c23740 4 client.5865 mds_command: new command op to 5589 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7a1c23740 1 -- 172.21.15.28:0/783759569 --> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574e0391350 con 0x5574e003be40 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe793fff640 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5574e003be40 0x5574e0480c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe7737fe640 10 client.5865 ms_handle_connect on v2:172.21.15.28:6836/1842070529 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.470+0000 7fe793fff640 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5574e003be40 0x5574e0480c70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be0045ec0fdf0bf3 server_cookie=3e4789a91c204aae in_seq=0 out_seq=0 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7737fe640 1 -- 172.21.15.28:0/783759569 <== mds.0 v2:172.21.15.28:6836/1842070529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5574e0391350 con 0x5574e003be40 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7737fe640 10 client.5865 handle_command_reply: tid=0 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 client.5865 shutdown 2024-04-25T19:13:15.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 -- 172.21.15.28:0/783759569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 msgr2=0x5574e04432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e04432f0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe78802a1e0 tx=0x7fe788007000 comp rx=0 tx=0).stop 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 -- 172.21.15.28:0/783759569 shutdown_connections 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5574e003be40 0x5574e0480c70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e007d270 0x5574e044b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e005c590 0x5574e0443830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7737fe640 0 client.5865 ms_handle_reset on v2:172.21.15.28:6836/1842070529 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/783759569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e04432f0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 -- 172.21.15.28:0/783759569 >> 172.21.15.28:0/783759569 conn(0x5574e00713c0 msgr2=0x5574e041dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 -- 172.21.15.28:0/783759569 shutdown_connections 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.471+0000 7fe7a1c23740 1 -- 172.21.15.28:0/783759569 wait complete. 2024-04-25T19:13:15.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.472+0000 7fe7a1c23740 20 client.5865 trim_cache size 0 max 16384 2024-04-25T19:13:15.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.490+0000 7fe7a1c23740 1 Processor -- start 2024-04-25T19:13:15.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.490+0000 7fe7a1c23740 1 -- start start 2024-04-25T19:13:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e0463620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e005c590 0x5574e0466cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 0x5574e0467230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7942034c0 con 0x5574e00543b0 2024-04-25T19:13:15.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe7941f70a0 con 0x5574e005c590 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe7941f76f0 con 0x5574e007d270 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e0463620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e0463620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59468/0 (socket says 172.21.15.28:59468) 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 -- 172.21.15.28:0/1889052820 learned_addr learned my addr 172.21.15.28:0/1889052820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe793fff640 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e005c590 0x5574e0466cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe799233640 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 0x5574e0467230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 -- 172.21.15.28:0/1889052820 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 msgr2=0x5574e0467230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 0x5574e0467230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 -- 172.21.15.28:0/1889052820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e005c590 msgr2=0x5574e0466cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e005c590 0x5574e0466cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 -- 172.21.15.28:0/1889052820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574e041a1b0 con 0x5574e00543b0 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe798a32640 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e0463620 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe7880b40a0 tx=0x7fe78804b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e8e1fc18f56dbab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe77dffb640 1 -- 172.21.15.28:0/1889052820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78807f460 con 0x5574e00543b0 2024-04-25T19:13:15.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe77dffb640 1 -- 172.21.15.28:0/1889052820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe788086040 con 0x5574e00543b0 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.492+0000 7fe77dffb640 1 -- 172.21.15.28:0/1889052820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7880887c0 con 0x5574e00543b0 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1889052820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 msgr2=0x5574e0463620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e0463620 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe7880b40a0 tx=0x7fe78804b000 comp rx=0 tx=0).stop 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1889052820 shutdown_connections 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 0x5574e0467230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e005c590 0x5574e0466cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1889052820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e00543b0 0x5574e0463620 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1889052820 >> 172.21.15.28:0/1889052820 conn(0x5574e00713c0 msgr2=0x5574e0467d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1889052820 shutdown_connections 2024-04-25T19:13:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.493+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1889052820 wait complete. 2024-04-25T19:13:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.494+0000 7fe7a1c23740 1 -- start start 2024-04-25T19:13:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.494+0000 7fe7a1c23740 1 Processor -- start 2024-04-25T19:13:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e04657f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e005c590 0x5574e0445b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7a1c23740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 0x5574e0446040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5574e015f6b0 con 0x5574e005c590 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5574e0383740 con 0x5574e00543b0 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7a1c23740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x5574e0443100 con 0x5574e007d270 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7a1c23740 10 client.0 _ll_register_callbacks cb 0x7fe78807bc40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e005c590 0x5574e0445b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e005c590 0x5574e0445b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59470/0 (socket says 172.21.15.28:59470) 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 -- 172.21.15.28:0/1090271247 learned_addr learned my addr 172.21.15.28:0/1090271247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe798a32640 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e04657f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe799233640 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 0x5574e0446040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 -- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 msgr2=0x5574e0446040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 0x5574e0446040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 -- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 msgr2=0x5574e04657f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e04657f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 -- 172.21.15.28:0/1090271247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574e0465d30 con 0x5574e005c590 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe798a32640 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e04657f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe793fff640 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e005c590 0x5574e0445b00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe7940c8f60 tx=0x7fe780004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4702acf2e547a6ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7737fe640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7737fe640 1 -- 172.21.15.28:0/1090271247 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7800108a0 con 0x5574e005c590 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7737fe640 1 -- 172.21.15.28:0/1090271247 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fe78000f040 con 0x5574e005c590 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.496+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1090271247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe788086040 con 0x5574e005c590 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7737fe640 1 -- 172.21.15.28:0/1090271247 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fe780039050 con 0x5574e005c590 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7a1c23740 10 client.5871 fetch_fsmap learned FSMap version 30 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1090271247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5574e04437a0 con 0x5574e005c590 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7737fe640 1 -- 172.21.15.28:0/1090271247 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe788086040 con 0x5574e005c590 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7737fe640 1 -- 172.21.15.28:0/1090271247 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 4139+0+0 (secure 0 0 0) 0x7fe78003c020 con 0x5574e005c590 2024-04-25T19:13:15.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7a1c23740 10 client.5871 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7a1c23740 10 client.5871 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7a1c23740 20 client.5871 populate_metadata read hostname 'smithi028' 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5574e0440fb0 0x5574e04413c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7a1c23740 4 client.5871 mds_command: new command op to 5589 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1090271247 --> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] -- command(tid 0: {"prefix": "status"}) -- 0x5574e038b1c0 con 0x5574e0440fb0 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe798a32640 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5574e0440fb0 0x5574e04413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.497+0000 7fe7737fe640 10 client.5871 ms_handle_connect on v2:172.21.15.28:6836/1842070529 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe798a32640 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5574e0440fb0 0x5574e04413c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=605ee3d56a0585aa server_cookie=a312c5223e50164a in_seq=0 out_seq=0 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7737fe640 1 -- 172.21.15.28:0/1090271247 <== mds.0 v2:172.21.15.28:6836/1842070529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x5574e038b1c0 con 0x5574e0440fb0 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7737fe640 10 client.5871 handle_command_reply: tid=0 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 client.5871 shutdown 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1090271247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e005c590 msgr2=0x5574e0445b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e005c590 0x5574e0445b00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe7940c8f60 tx=0x7fe780004730 comp rx=0 tx=0).stop 2024-04-25T19:13:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1090271247 shutdown_connections 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5574e0440fb0 0x5574e04413c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5574e007d270 0x5574e0446040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574e005c590 0x5574e0445b00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 --2- 172.21.15.28:0/1090271247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5574e00543b0 0x5574e04657f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1090271247 >> 172.21.15.28:0/1090271247 conn(0x5574e00713c0 msgr2=0x5574e0467d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7737fe640 0 client.5871 ms_handle_reset on v2:172.21.15.28:6836/1842070529 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1090271247 shutdown_connections 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 1 -- 172.21.15.28:0/1090271247 wait complete. 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.498+0000 7fe7a1c23740 20 client.5871 trim_cache size 0 max 16384 2024-04-25T19:13:15.506 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "5d8007d4-24a5-41be-aa4d-cd64fedac62a", 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5589, 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "recovery_fs", 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.5248675880000002, 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 30, 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 48, 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 48, 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 14.454651535 2024-04-25T19:13:15.507 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:15.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 -- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe7780744e0 msgr2=0x7fe7780769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe7780744e0 0x7fe7780769f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe7940d3520 tx=0x7fe780008040 comp rx=0 tx=0).stop 2024-04-25T19:13:15.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 -- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe79408cce0 msgr2=0x7fe7941f94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe79408cce0 0x7fe7941f94a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe788000c00 tx=0x7fe7880046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 -- 172.21.15.28:0/3995040662 shutdown_connections 2024-04-25T19:13:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe7780744e0 0x7fe7780769f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe794160310 0x7fe7940cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7940b79f0 0x7fe7941f9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 --2- 172.21.15.28:0/3995040662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe79408cce0 0x7fe7941f94a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 -- 172.21.15.28:0/3995040662 >> 172.21.15.28:0/3995040662 conn(0x7fe7940d2a50 msgr2=0x7fe7940d63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 -- 172.21.15.28:0/3995040662 shutdown_connections 2024-04-25T19:13:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.499+0000 7fe79ac96640 1 -- 172.21.15.28:0/3995040662 wait complete. 2024-04-25T19:13:15.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 20 2024-04-25T19:13:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.625+0000 7fe1469e3640 1 Processor -- start 2024-04-25T19:13:15.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.625+0000 7fe1469e3640 1 -- start start 2024-04-25T19:13:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.626+0000 7fe1469e3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1400b3d00 0x7fe1400b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.626+0000 7fe1469e3640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe1400b4650 0x7fe14016a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.626+0000 7fe1469e3640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1400b92a0 con 0x7fe1400d2d10 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.626+0000 7fe1469e3640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fe1400d28d0 con 0x7fe1400b4650 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.626+0000 7fe1469e3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe14006e690 con 0x7fe1400b3d00 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13ffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1400d2d10 0x7fe1400d21c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36954/0 (socket says 172.21.15.28:36954) 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13ffff640 1 -- 172.21.15.28:0/2401153448 learned_addr learned my addr 172.21.15.28:0/2401153448 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe144f80640 1 --2- 172.21.15.28:0/2401153448 >> v2:172.21.15.53:3300/0 conn(0x7fe1400b4650 0x7fe14016a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13f7fe640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1400b3d00 0x7fe1400b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1023750880 0 0) 0x7fe1400b92a0 con 0x7fe1400d2d10 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe128003520 con 0x7fe1400d2d10 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1931019806 0 0) 0x7fe128003520 con 0x7fe1400d2d10 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1400b3d00 msgr2=0x7fe1400b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1400b3d00 0x7fe1400b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 >> v2:172.21.15.53:3300/0 conn(0x7fe1400b4650 msgr2=0x7fe14016a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 --2- 172.21.15.28:0/2401153448 >> v2:172.21.15.53:3300/0 conn(0x7fe1400b4650 0x7fe14016a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe14009cea0 con 0x7fe1400d2d10 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fe134001670 con 0x7fe1400d2d10 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe144f80640 1 --2- 172.21.15.28:0/2401153448 >> v2:172.21.15.53:3300/0 conn(0x7fe1400b4650 0x7fe14016a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 >> v1:172.21.15.28:6789/0 conn(0x7fe1400d2d10 legacy=0x7fe1400d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1400b3d00 0x7fe1280039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe128003f30 0x7fe128004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe128004d40 0x7fe128005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1400d28d0 con 0x7fe128003f30 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe14006e690 con 0x7fe128004d40 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe12800bac0 con 0x7fe1400b3d00 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.627+0000 7fe13f7fe640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe128003f30 0x7fe128004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.628+0000 7fe13ffff640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1400b3d00 0x7fe1280039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.628+0000 7fe144f80640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe128004d40 0x7fe128005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.628+0000 7fe13f7fe640 1 -- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1400b3d00 msgr2=0x7fe1280039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.628+0000 7fe13f7fe640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe1400b3d00 0x7fe1280039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.628+0000 7fe13f7fe640 1 -- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe128004d40 msgr2=0x7fe128005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.628+0000 7fe13f7fe640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe128004d40 0x7fe128005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.628+0000 7fe13f7fe640 1 -- 172.21.15.28:0/2401153448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe128003520 con 0x7fe128003f30 2024-04-25T19:13:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe13f7fe640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe128003f30 0x7fe128004800 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe1400d0420 tx=0x7fe128005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b509b2dc4b575021 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130036070 con 0x7fe128003f30 2024-04-25T19:13:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe130031400 con 0x7fe128003f30 2024-04-25T19:13:15.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe13effd640 1 -- 172.21.15.28:0/2401153448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130009690 con 0x7fe128003f30 2024-04-25T19:13:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe1469e3640 1 -- 172.21.15.28:0/2401153448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe128003f30 msgr2=0x7fe128004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe1469e3640 1 --2- 172.21.15.28:0/2401153448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe128003f30 0x7fe128004800 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fe1400d0420 tx=0x7fe128005830 comp rx=0 tx=0).stop 2024-04-25T19:13:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe144f80640 1 -- 172.21.15.28:0/2401153448 reap_dead start 2024-04-25T19:13:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe1469e3640 1 -- 172.21.15.28:0/2401153448 shutdown_connections 2024-04-25T19:13:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe1469e3640 1 -- 172.21.15.28:0/2401153448 >> 172.21.15.28:0/2401153448 conn(0x7fe1400d0450 msgr2=0x7fe1400d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe1469e3640 1 -- 172.21.15.28:0/2401153448 shutdown_connections 2024-04-25T19:13:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.629+0000 7fe1469e3640 1 -- 172.21.15.28:0/2401153448 wait complete. 2024-04-25T19:13:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.631+0000 7fe1469e3640 1 Processor -- start 2024-04-25T19:13:15.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.631+0000 7fe1469e3640 1 -- start start 2024-04-25T19:13:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe1469e3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe128003f30 0x7fe1400c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe1469e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1400b1880 0x7fe1400b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe1469e3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1401d18f0 0x7fe1401d1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe1469e3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe14006ebc0 con 0x7fe1400b1880 2024-04-25T19:13:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe1469e3640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fe1400b8ee0 con 0x7fe1401d18f0 2024-04-25T19:13:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe1469e3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fe1400d3120 con 0x7fe128003f30 2024-04-25T19:13:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1400b1880 0x7fe1400b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1400b1880 0x7fe1400b1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59486/0 (socket says 172.21.15.28:59486) 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 -- 172.21.15.28:0/1344474682 learned_addr learned my addr 172.21.15.28:0/1344474682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13ffff640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe128003f30 0x7fe1400c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe144f80640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1401d18f0 0x7fe1401d1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 -- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe128003f30 msgr2=0x7fe1400c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe128003f30 0x7fe1400c7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 -- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1401d18f0 msgr2=0x7fe1401d1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1401d18f0 0x7fe1401d1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 -- 172.21.15.28:0/1344474682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1400b6690 con 0x7fe1400b1880 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13ffff640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe128003f30 0x7fe1400c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe144f80640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1401d18f0 0x7fe1401d1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.633+0000 7fe13f7fe640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1400b1880 0x7fe1400b1c90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe130034040 tx=0x7fe130004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.634+0000 7fe13cff9640 1 -- 172.21.15.28:0/1344474682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130040020 con 0x7fe1400b1880 2024-04-25T19:13:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.634+0000 7fe13cff9640 1 -- 172.21.15.28:0/1344474682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe13003b6a0 con 0x7fe1400b1880 2024-04-25T19:13:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.634+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1401d2240 con 0x7fe1400b1880 2024-04-25T19:13:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.634+0000 7fe13cff9640 1 -- 172.21.15.28:0/1344474682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130036040 con 0x7fe1400b1880 2024-04-25T19:13:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.634+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1400b6c50 con 0x7fe1400b1880 2024-04-25T19:13:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.635+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe14006d220 con 0x7fe1400b1880 2024-04-25T19:13:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.635+0000 7fe13cff9640 1 -- 172.21.15.28:0/1344474682 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe130043020 con 0x7fe1400b1880 2024-04-25T19:13:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.636+0000 7fe13cff9640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe12c0744b0 0x7fe12c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.636+0000 7fe13cff9640 1 -- 172.21.15.28:0/1344474682 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fe1300b5bf0 con 0x7fe1400b1880 2024-04-25T19:13:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.636+0000 7fe13ffff640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe12c0744b0 0x7fe12c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.636+0000 7fe13ffff640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe12c0744b0 0x7fe12c0769c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe140075cb0 tx=0x7fe134009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.641+0000 7fe13cff9640 1 -- 172.21.15.28:0/1344474682 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe14006d220 con 0x7fe1400b1880 2024-04-25T19:13:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.827+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7fe1400c7d50 con 0x7fe1400b1880 2024-04-25T19:13:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.831+0000 7fe13cff9640 1 -- 172.21.15.28:0/1344474682 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v11)=0 v11) ==== 108+0+0 (secure 0 0 0) 0x7fe13004a070 con 0x7fe1400b1880 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.833+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe12c0744b0 msgr2=0x7fe12c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.833+0000 7fe1469e3640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe12c0744b0 0x7fe12c0769c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe140075cb0 tx=0x7fe134009040 comp rx=0 tx=0).stop 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.833+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1400b1880 msgr2=0x7fe1400b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.833+0000 7fe1469e3640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1400b1880 0x7fe1400b1c90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fe130034040 tx=0x7fe130004730 comp rx=0 tx=0).stop 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.834+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 shutdown_connections 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.834+0000 7fe1469e3640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fe12c0744b0 0x7fe12c0769c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.834+0000 7fe1469e3640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1401d18f0 0x7fe1401d1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.834+0000 7fe1469e3640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1400b1880 0x7fe1400b1c90 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.834+0000 7fe1469e3640 1 --2- 172.21.15.28:0/1344474682 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fe128003f30 0x7fe1400c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.834+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 >> 172.21.15.28:0/1344474682 conn(0x7fe1400d0450 msgr2=0x7fe1400d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.834+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 shutdown_connections 2024-04-25T19:13:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.834+0000 7fe1469e3640 1 -- 172.21.15.28:0/1344474682 wait complete. 2024-04-25T19:13:15.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / force,recursive,repair 2024-04-25T19:13:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.961+0000 7f9197572640 1 Processor -- start 2024-04-25T19:13:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.961+0000 7f9197572640 1 -- start start 2024-04-25T19:13:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.962+0000 7f9197572640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f91900b14a0 0x7f91900b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.962+0000 7f9197572640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91900b0de0 0x7f91900b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.962+0000 7f9197572640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9190164280 con 0x7f91901604e0 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.962+0000 7f9197572640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f91900b3630 con 0x7f91900b0de0 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.962+0000 7f9197572640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f9190164560 con 0x7f91900b14a0 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9195b0f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91900b0de0 0x7f91900b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9195b0f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f91900b0de0 0x7f91900b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59498/0 (socket says 172.21.15.28:59498) 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9195b0f640 1 -- 172.21.15.28:0/2665334999 learned_addr learned my addr 172.21.15.28:0/2665334999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9194b0d640 1 --2- 172.21.15.28:0/2665334999 >> v2:172.21.15.53:3300/0 conn(0x7f91900b14a0 0x7f91900b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9195b0f640 1 -- 172.21.15.28:0/2665334999 >> v2:172.21.15.53:3300/0 conn(0x7f91900b14a0 msgr2=0x7f91900b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9195b0f640 1 --2- 172.21.15.28:0/2665334999 >> v2:172.21.15.53:3300/0 conn(0x7f91900b14a0 0x7f91900b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9195b0f640 1 -- 172.21.15.28:0/2665334999 >> v1:172.21.15.53:6789/0 conn(0x7f91901604e0 legacy=0x7f91900d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9195b0f640 1 -- 172.21.15.28:0/2665334999 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9190164840 con 0x7f91900b0de0 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.963+0000 7f9194b0d640 1 --2- 172.21.15.28:0/2665334999 >> v2:172.21.15.53:3300/0 conn(0x7f91900b14a0 0x7f91900b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:15.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.964+0000 7f9195b0f640 1 --2- 172.21.15.28:0/2665334999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f91900b30f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9190074590 tx=0x7f918802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211e9e2fbdccad80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.964+0000 7f9187fff640 1 -- 172.21.15.28:0/2665334999 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188036070 con 0x7f91900b0de0 2024-04-25T19:13:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.964+0000 7f9187fff640 1 -- 172.21.15.28:0/2665334999 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9188031600 con 0x7f91900b0de0 2024-04-25T19:13:15.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.964+0000 7f9187fff640 1 -- 172.21.15.28:0/2665334999 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91880319d0 con 0x7f91900b0de0 2024-04-25T19:13:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.965+0000 7f9197572640 1 -- 172.21.15.28:0/2665334999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 msgr2=0x7f91900b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.965+0000 7f9197572640 1 --2- 172.21.15.28:0/2665334999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f91900b30f0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9190074590 tx=0x7f918802a730 comp rx=0 tx=0).stop 2024-04-25T19:13:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.965+0000 7f9197572640 1 -- 172.21.15.28:0/2665334999 shutdown_connections 2024-04-25T19:13:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.965+0000 7f9197572640 1 --2- 172.21.15.28:0/2665334999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f91900b30f0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.965+0000 7f9197572640 1 --2- 172.21.15.28:0/2665334999 >> v2:172.21.15.53:3300/0 conn(0x7f91900b14a0 0x7f91900b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.965+0000 7f9197572640 1 -- 172.21.15.28:0/2665334999 >> 172.21.15.28:0/2665334999 conn(0x7f91900b58b0 msgr2=0x7f91900b6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.965+0000 7f9197572640 1 -- 172.21.15.28:0/2665334999 shutdown_connections 2024-04-25T19:13:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.965+0000 7f9197572640 1 -- 172.21.15.28:0/2665334999 wait complete. 2024-04-25T19:13:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.967+0000 7f9197572640 1 Processor -- start 2024-04-25T19:13:15.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.967+0000 7f9197572640 1 -- start start 2024-04-25T19:13:15.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9197572640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f919015f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9197572640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91900b14a0 0x7f919009bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9197572640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91901604e0 0x7f9190151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9197572640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9190163f70 con 0x7f91900b0de0 2024-04-25T19:13:15.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9197572640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9190163df0 con 0x7f91901604e0 2024-04-25T19:13:15.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9197572640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9190160c50 con 0x7f91900b14a0 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f919530e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f919015f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f919530e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f919015f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59506/0 (socket says 172.21.15.28:59506) 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f919530e640 1 -- 172.21.15.28:0/3553381660 learned_addr learned my addr 172.21.15.28:0/3553381660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9195b0f640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91901604e0 0x7f9190151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9194b0d640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91900b14a0 0x7f919009bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f919530e640 1 -- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91900b14a0 msgr2=0x7f919009bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f919530e640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91900b14a0 0x7f919009bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f919530e640 1 -- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91901604e0 msgr2=0x7f9190151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f919530e640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91901604e0 0x7f9190151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f919530e640 1 -- 172.21.15.28:0/3553381660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91901524f0 con 0x7f91900b0de0 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.968+0000 7f9194b0d640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91900b14a0 0x7f919009bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.969+0000 7f919530e640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f919015f7c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f918c00c3c0 tx=0x7f918c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.969+0000 7f9185ffb640 1 -- 172.21.15.28:0/3553381660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c0052d0 con 0x7f91900b0de0 2024-04-25T19:13:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.969+0000 7f9185ffb640 1 -- 172.21.15.28:0/3553381660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918c005490 con 0x7f91900b0de0 2024-04-25T19:13:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.969+0000 7f9197572640 1 -- 172.21.15.28:0/3553381660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f919009c510 con 0x7f91900b0de0 2024-04-25T19:13:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.969+0000 7f9185ffb640 1 -- 172.21.15.28:0/3553381660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c007260 con 0x7f91900b0de0 2024-04-25T19:13:15.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.969+0000 7f9197572640 1 -- 172.21.15.28:0/3553381660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f919009cad0 con 0x7f91900b0de0 2024-04-25T19:13:15.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.971+0000 7f9185ffb640 1 -- 172.21.15.28:0/3553381660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f918c007400 con 0x7f91900b0de0 2024-04-25T19:13:15.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.971+0000 7f9185ffb640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91740744b0 0x7f91740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.972+0000 7f9185ffb640 1 -- 172.21.15.28:0/3553381660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f918c094a00 con 0x7f91900b0de0 2024-04-25T19:13:15.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.972+0000 7f9194b0d640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91740744b0 0x7f91740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.972+0000 7f9194b0d640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91740744b0 0x7f91740769c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9190152db0 tx=0x7f9180008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.972+0000 7f919e424740 1 Processor -- start 2024-04-25T19:13:15.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.972+0000 7f919e424740 1 -- start start 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919e424740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca16bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919e424740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca16f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919e424740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca19baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919e424740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570ca0d6a10 con 0x5570c9daf7f0 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919e424740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5570ca0727d0 con 0x5570c9da7610 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919e424740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x5570ca0d29d0 con 0x5570c9dd04d0 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f9195b0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca19baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f9195b0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca19baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59510/0 (socket says 172.21.15.28:59510) 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f9195b0f640 1 -- 172.21.15.28:0/1812096955 learned_addr learned my addr 172.21.15.28:0/1812096955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919530e640 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca16bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f9194b0d640 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca16f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919530e640 1 -- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 msgr2=0x5570ca16f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919530e640 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca16f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919530e640 1 -- 172.21.15.28:0/1812096955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 msgr2=0x5570ca19baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919530e640 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca19baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919530e640 1 -- 172.21.15.28:0/1812096955 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570c9eb4940 con 0x5570c9dd04d0 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f9195b0f640 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca19baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.974+0000 7f919530e640 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca16bc50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f918c092bc0 tx=0x7f918c000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7e1355725689fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f917affd640 1 -- 172.21.15.28:0/1812096955 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c067530 con 0x5570c9dd04d0 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f917affd640 1 -- 172.21.15.28:0/1812096955 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918c05aa10 con 0x5570c9dd04d0 2024-04-25T19:13:15.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f917affd640 1 -- 172.21.15.28:0/1812096955 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918c05dc20 con 0x5570c9dd04d0 2024-04-25T19:13:15.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f919e424740 1 -- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 msgr2=0x5570ca16bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f919e424740 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca16bc50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f918c092bc0 tx=0x7f918c000ed0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f919e424740 1 -- 172.21.15.28:0/1812096955 shutdown_connections 2024-04-25T19:13:15.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f919e424740 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca19baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f919e424740 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca16f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.975+0000 7f919e424740 1 --2- 172.21.15.28:0/1812096955 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca16bc50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.976+0000 7f919e424740 1 -- 172.21.15.28:0/1812096955 >> 172.21.15.28:0/1812096955 conn(0x5570c9dc4620 msgr2=0x5570ca171170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.976+0000 7f919e424740 1 -- 172.21.15.28:0/1812096955 shutdown_connections 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.976+0000 7f919e424740 1 -- 172.21.15.28:0/1812096955 wait complete. 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.976+0000 7f919e424740 1 -- start start 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.977+0000 7f919e424740 1 Processor -- start 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f919e424740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca18f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f919e424740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 0x5570ca19d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f919e424740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca1af000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f919e424740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5570ca199560 con 0x5570c9dd04d0 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f919e424740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5570ca0704a0 con 0x5570c9da7610 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f919e424740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x5570ca16c190 con 0x5570c9daf7f0 2024-04-25T19:13:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f919e424740 10 client.0 _ll_register_callbacks cb 0x5570ca07edd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9195b0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca1af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9195b0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca1af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59520/0 (socket says 172.21.15.28:59520) 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9195b0f640 1 -- 172.21.15.28:0/1569736948 learned_addr learned my addr 172.21.15.28:0/1569736948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f919530e640 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca18f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9195b0f640 1 -- 172.21.15.28:0/1569736948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 msgr2=0x5570ca19d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9194b0d640 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 0x5570ca19d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9195b0f640 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 0x5570ca19d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9195b0f640 1 -- 172.21.15.28:0/1569736948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 msgr2=0x5570ca18f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9195b0f640 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca18f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.978+0000 7f9195b0f640 1 -- 172.21.15.28:0/1569736948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570c9f150d0 con 0x5570c9dd04d0 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f9195b0f640 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca1af000 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9188031b70 tx=0x7f918802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1defd2f060446b61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f916ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f916ffff640 1 -- 172.21.15.28:0/1569736948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188042070 con 0x5570c9dd04d0 2024-04-25T19:13:15.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f916ffff640 1 -- 172.21.15.28:0/1569736948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f918803e470 con 0x5570c9dd04d0 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 1 -- 172.21.15.28:0/1569736948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f918c05aa10 con 0x5570c9dd04d0 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f916ffff640 1 -- 172.21.15.28:0/1569736948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f918804c050 con 0x5570c9dd04d0 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 10 client.5889 fetch_fsmap learned FSMap version 30 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 1 -- 172.21.15.28:0/1569736948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5570c9da8230 con 0x5570c9dd04d0 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f916ffff640 1 -- 172.21.15.28:0/1569736948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918c05aa10 con 0x5570c9dd04d0 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f916ffff640 1 -- 172.21.15.28:0/1569736948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 4139+0+0 (secure 0 0 0) 0x7f918804f020 con 0x5570c9dd04d0 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 10 client.5889 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 10 client.5889 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 20 client.5889 populate_metadata read hostname 'smithi028' 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5570c9d8f2b0 0x5570ca1b0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 4 client.5889 mds_command: new command op to 5589 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.979+0000 7f919e424740 1 -- 172.21.15.28:0/1569736948 --> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570ca0aa3a0 con 0x5570c9d8f2b0 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.980+0000 7f919530e640 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5570c9d8f2b0 0x5570ca1b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:15.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.980+0000 7f916ffff640 10 client.5889 ms_handle_connect on v2:172.21.15.28:6836/1842070529 2024-04-25T19:13:15.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.980+0000 7f919530e640 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5570c9d8f2b0 0x5570ca1b0360 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fe90ed7fbfb4f4 server_cookie=c0ba1ed6f3aabd in_seq=0 out_seq=0 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f916ffff640 1 -- 172.21.15.28:0/1569736948 <== mds.0 v2:172.21.15.28:6836/1842070529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5570ca0aa3a0 con 0x5570c9d8f2b0 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f916ffff640 10 client.5889 handle_command_reply: tid=0 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 client.5889 shutdown 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 -- 172.21.15.28:0/1569736948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 msgr2=0x5570ca1af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca1af000 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9188031b70 tx=0x7f918802c000 comp rx=0 tx=0).stop 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 -- 172.21.15.28:0/1569736948 shutdown_connections 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5570c9d8f2b0 0x5570ca1b0360 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca1af000 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 0x5570ca19d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 --2- 172.21.15.28:0/1569736948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca18f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:15.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f916ffff640 0 client.5889 ms_handle_reset on v2:172.21.15.28:6836/1842070529 2024-04-25T19:13:15.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 -- 172.21.15.28:0/1569736948 >> 172.21.15.28:0/1569736948 conn(0x5570c9dc4620 msgr2=0x5570ca19baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:15.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 -- 172.21.15.28:0/1569736948 shutdown_connections 2024-04-25T19:13:15.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 1 -- 172.21.15.28:0/1569736948 wait complete. 2024-04-25T19:13:15.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:15.981+0000 7f919e424740 20 client.5889 trim_cache size 0 max 16384 2024-04-25T19:13:16.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.000+0000 7f919e424740 1 Processor -- start 2024-04-25T19:13:16.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.000+0000 7f919e424740 1 -- start start 2024-04-25T19:13:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f919e424740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca18ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f919e424740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca1c8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f919e424740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca1c8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f919e424740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9190204dd0 con 0x5570c9daf7f0 2024-04-25T19:13:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f919e424740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9190203490 con 0x5570c9da7610 2024-04-25T19:13:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f919e424740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f91902027f0 con 0x5570c9dd04d0 2024-04-25T19:13:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f9194b0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca1c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f9194b0d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca1c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59532/0 (socket says 172.21.15.28:59532) 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f9194b0d640 1 -- 172.21.15.28:0/1811998331 learned_addr learned my addr 172.21.15.28:0/1811998331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f919530e640 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca18ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.001+0000 7f9195b0f640 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca1c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f9194b0d640 1 -- 172.21.15.28:0/1811998331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 msgr2=0x5570ca1c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f9194b0d640 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca1c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f9194b0d640 1 -- 172.21.15.28:0/1811998331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 msgr2=0x5570ca18ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f9194b0d640 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca18ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f9194b0d640 1 -- 172.21.15.28:0/1811998331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570ca1c5fd0 con 0x5570c9daf7f0 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f9194b0d640 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca1c8410 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9180006fd0 tx=0x7f9180015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e464a9278af161 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f917a7fc640 1 -- 172.21.15.28:0/1811998331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f918001b040 con 0x5570c9daf7f0 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f917a7fc640 1 -- 172.21.15.28:0/1811998331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918001c040 con 0x5570c9daf7f0 2024-04-25T19:13:16.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.002+0000 7f917a7fc640 1 -- 172.21.15.28:0/1811998331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9180014de0 con 0x5570c9daf7f0 2024-04-25T19:13:16.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 -- 172.21.15.28:0/1811998331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 msgr2=0x5570ca1c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca1c8410 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9180006fd0 tx=0x7f9180015000 comp rx=0 tx=0).stop 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 -- 172.21.15.28:0/1811998331 shutdown_connections 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9dd04d0 0x5570ca1c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9daf7f0 0x5570ca1c8410 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 --2- 172.21.15.28:0/1811998331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca18ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 -- 172.21.15.28:0/1811998331 >> 172.21.15.28:0/1811998331 conn(0x5570c9dc4620 msgr2=0x5570ca16f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 -- 172.21.15.28:0/1811998331 shutdown_connections 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.003+0000 7f919e424740 1 -- 172.21.15.28:0/1811998331 wait complete. 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.004+0000 7f919e424740 1 -- start start 2024-04-25T19:13:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.004+0000 7f919e424740 1 Processor -- start 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f919e424740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca22a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f919e424740 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 0x5570ca2338c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f919e424740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca237670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f919e424740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9190202e40 con 0x5570c9dd04d0 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f919e424740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x5570ca234390 con 0x5570c9da7610 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f919e424740 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x5570ca170710 con 0x5570c9daf7f0 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f919e424740 10 client.0 _ll_register_callbacks cb 0x7f9188009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f9195b0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca237670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f9195b0f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca237670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59544/0 (socket says 172.21.15.28:59544) 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.005+0000 7f9195b0f640 1 -- 172.21.15.28:0/3103761346 learned_addr learned my addr 172.21.15.28:0/3103761346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f919530e640 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca22a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f9195b0f640 1 -- 172.21.15.28:0/3103761346 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 msgr2=0x5570ca2338c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f9194b0d640 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 0x5570ca2338c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f9195b0f640 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 0x5570ca2338c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f9195b0f640 1 -- 172.21.15.28:0/3103761346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 msgr2=0x5570ca22a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f9195b0f640 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca22a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f9195b0f640 1 -- 172.21.15.28:0/3103761346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570ca170ba0 con 0x5570c9dd04d0 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f9195b0f640 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca237670 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9188031b70 tx=0x7f9188004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2977620013d3938 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f916ffff640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f916ffff640 1 -- 172.21.15.28:0/3103761346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188054020 con 0x5570c9dd04d0 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f916ffff640 1 -- 172.21.15.28:0/3103761346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f918803ec90 con 0x5570c9dd04d0 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f919e424740 1 -- 172.21.15.28:0/3103761346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f918001c040 con 0x5570c9dd04d0 2024-04-25T19:13:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f916ffff640 1 -- 172.21.15.28:0/3103761346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f918805b050 con 0x5570c9dd04d0 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f919e424740 10 client.5895 fetch_fsmap learned FSMap version 30 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f919e424740 1 -- 172.21.15.28:0/3103761346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5570ca1d04d0 con 0x5570c9dd04d0 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.006+0000 7f916ffff640 1 -- 172.21.15.28:0/3103761346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f918001c040 con 0x5570c9dd04d0 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f916ffff640 1 -- 172.21.15.28:0/3103761346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 30) ==== 4139+0+0 (secure 0 0 0) 0x7f918805e020 con 0x5570c9dd04d0 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f919e424740 10 client.5895 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f919e424740 10 client.5895 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f919e424740 20 client.5895 populate_metadata read hostname 'smithi028' 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f919e424740 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5570ca1d9830 0x5570ca1d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f919e424740 4 client.5895 mds_command: new command op to 5589 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive", "repair"]}] 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f919e424740 1 -- 172.21.15.28:0/3103761346 --> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive", "repair"]}) -- 0x5570ca0e4620 con 0x5570ca1d9830 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f919530e640 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5570ca1d9830 0x5570ca1d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f916ffff640 10 client.5895 ms_handle_connect on v2:172.21.15.28:6836/1842070529 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.007+0000 7f919530e640 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5570ca1d9830 0x5570ca1d8c70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2a3ed64a5e064b1 server_cookie=8d253867d8a1e263 in_seq=0 out_seq=0 2024-04-25T19:13:16.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f916ffff640 1 -- 172.21.15.28:0/3103761346 <== mds.0 v2:172.21.15.28:6836/1842070529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5570ca0e4620 con 0x5570ca1d9830 2024-04-25T19:13:16.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f916ffff640 10 client.5895 handle_command_reply: tid=0 2024-04-25T19:13:16.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 client.5895 shutdown 2024-04-25T19:13:16.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 -- 172.21.15.28:0/3103761346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 msgr2=0x5570ca237670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca237670 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9188031b70 tx=0x7f9188004ab0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 -- 172.21.15.28:0/3103761346 shutdown_connections 2024-04-25T19:13:16.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.28:6836/1842070529,v1:172.21.15.28:6837/1842070529] conn(0x5570ca1d9830 0x5570ca1d8c70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f916ffff640 0 client.5895 ms_handle_reset on v2:172.21.15.28:6836/1842070529 2024-04-25T19:13:16.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5570c9dd04d0 0x5570ca237670 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x5570c9daf7f0 0x5570ca2338c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 --2- 172.21.15.28:0/3103761346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5570c9da7610 0x5570ca22a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 -- 172.21.15.28:0/3103761346 >> 172.21.15.28:0/3103761346 conn(0x5570c9dc4620 msgr2=0x5570ca16f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:16.015 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f95c5b58640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000000 2024-04-25T19:13:16.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 -- 172.21.15.28:0/3103761346 shutdown_connections 2024-04-25T19:13:16.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.008+0000 7f919e424740 1 -- 172.21.15.28:0/3103761346 wait complete. 2024-04-25T19:13:16.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.009+0000 7f919e424740 20 client.5895 trim_cache size 0 max 16384 2024-04-25T19:13:16.016 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-25T19:13:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "fd077573-60a4-4379-a9aa-1eac94c63b84", 2024-04-25T19:13:16.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-25T19:13:16.016 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:16.017 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-04-25T19:13:16.009+0000 7f95c5b58640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /subdir/ auth v1 pv4 ap=3 DIRTYPARENT f(v0 1=1+0) n() (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55bea449d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000000:[<0x1/subdir v1>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=1+0)","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 1=1+0)","memory_value.rstat":"n()","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.009+0000 7f9197572640 1 -- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91740744b0 msgr2=0x7f91740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.009+0000 7f9197572640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91740744b0 0x7f91740769c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9190152db0 tx=0x7f9180008040 comp rx=0 tx=0).stop 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.009+0000 7f9197572640 1 -- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 msgr2=0x7f919015f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.009+0000 7f9197572640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f919015f7c0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f918c00c3c0 tx=0x7f918c00c880 comp rx=0 tx=0).stop 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.010+0000 7f9197572640 1 -- 172.21.15.28:0/3553381660 shutdown_connections 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.010+0000 7f9197572640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f91740744b0 0x7f91740769c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.010+0000 7f9197572640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f91901604e0 0x7f9190151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.010+0000 7f9197572640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f91900b14a0 0x7f919009bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.010+0000 7f9197572640 1 --2- 172.21.15.28:0/3553381660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91900b0de0 0x7f919015f7c0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.010+0000 7f9197572640 1 -- 172.21.15.28:0/3553381660 >> 172.21.15.28:0/3553381660 conn(0x7f91900b58b0 msgr2=0x7f91900b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:16.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.010+0000 7f9197572640 1 -- 172.21.15.28:0/3553381660 shutdown_connections 2024-04-25T19:13:16.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.010+0000 7f9197572640 1 -- 172.21.15.28:0/3553381660 wait complete. 2024-04-25T19:13:16.019 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-04-25T19:13:16.012+0000 7f95c5b58640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /subdir/sixmegs auth v1 ap=1 DIRTYPARENT s=6291456 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x55bea4514100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(6)0x10000000001:[<0x10000000000/sixmegs v10>,<0x1/subdir v12>]//[]","memoryvalue":"(6)0x10000000001:[<0x10000000000/sixmegs v1>,<0x1/subdir v4>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T19:13:16.019 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-04-25T19:13:16.014+0000 7f95c5b58640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1 [...2,head] / auth v4 pv6 ap=5 snaprealm=0x55bea4498480 DIRTYPARENT f(v1 1=0+1) n(v0 1=0+1)/n() (inest mix->lock(2) w=1 flushing) (ifile mix->sync w=1 flushing) | dirtyscattered=2 lock=2 dirfrag=1 openingsnapparents=0 dirtyparent=1 dirty=1 waiter=1 authpin=1 scrubqueue=0 0x55bea449cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x1:[]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=0+1)","ondisk_value.rstat":"n(v0 2=0+2)","memory_value.dirstat":"f(v1 1=0+1)","memory_value.rstat":"n(v0 1=0+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-04-25T19:13:16.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.133+0000 7fbb384af640 1 Processor -- start 2024-04-25T19:13:16.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.133+0000 7fbb384af640 1 -- start start 2024-04-25T19:13:16.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb384af640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb300cf120 0x7fbb300d2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb384af640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb300cf530 con 0x7fbb3009d010 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb384af640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb3006d710 con 0x7fbb3009cb60 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb384af640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3008cc20 con 0x7fbb300cf120 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb3624b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbb3009d010 0x7fbb3006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36970/0 (socket says 172.21.15.28:36970) 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb3624b640 1 -- 172.21.15.28:0/3269724580 learned_addr learned my addr 172.21.15.28:0/3269724580 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb36a4c640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb300cf120 0x7fbb300d2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2603734713 0 0) 0x7fbb300cf530 con 0x7fbb3009d010 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb18003520 con 0x7fbb3009d010 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653914594 0 0) 0x7fbb3006d710 con 0x7fbb3009cb60 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb300cf530 con 0x7fbb3009cb60 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.135+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2248820749 0 0) 0x7fbb18003520 con 0x7fbb3009d010 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb300cf120 msgr2=0x7fbb300d2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb300cf120 0x7fbb300d2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 >> v1:172.21.15.53:6789/0 conn(0x7fbb3009cb60 legacy=0x7fbb300cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3008ce70 con 0x7fbb3009d010 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fbb280017f0 con 0x7fbb3009d010 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 >> v1:172.21.15.28:6789/0 conn(0x7fbb3009d010 legacy=0x7fbb3006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18003b70 0x7fbb18003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb300cf120 0x7fbb18004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb18004d40 0x7fbb180051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3008cc20 con 0x7fbb18004d40 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1800bb50 con 0x7fbb18003b70 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1800bd50 con 0x7fbb300cf120 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb36a4c640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb18004d40 0x7fbb180051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb3624b640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18003b70 0x7fbb18003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb35a4a640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb300cf120 0x7fbb18004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb36a4c640 1 -- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb300cf120 msgr2=0x7fbb18004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb36a4c640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb300cf120 0x7fbb18004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb36a4c640 1 -- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18003b70 msgr2=0x7fbb18003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb36a4c640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18003b70 0x7fbb18003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.136+0000 7fbb36a4c640 1 -- 172.21.15.28:0/3269724580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb18003520 con 0x7fbb18004d40 2024-04-25T19:13:16.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.137+0000 7fbb36a4c640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb18004d40 0x7fbb180051b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbb300d1370 tx=0x7fbb18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7592526bdcacd99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:16.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.137+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20009c10 con 0x7fbb18004d40 2024-04-25T19:13:16.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.137+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb20036070 con 0x7fbb18004d40 2024-04-25T19:13:16.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.137+0000 7fbb35249640 1 -- 172.21.15.28:0/3269724580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb200317f0 con 0x7fbb18004d40 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.138+0000 7fbb384af640 1 -- 172.21.15.28:0/3269724580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb18004d40 msgr2=0x7fbb180051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.138+0000 7fbb384af640 1 --2- 172.21.15.28:0/3269724580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb18004d40 0x7fbb180051b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fbb300d1370 tx=0x7fbb18005830 comp rx=0 tx=0).stop 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.138+0000 7fbb36a4c640 1 -- 172.21.15.28:0/3269724580 reap_dead start 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.138+0000 7fbb384af640 1 -- 172.21.15.28:0/3269724580 shutdown_connections 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.138+0000 7fbb384af640 1 -- 172.21.15.28:0/3269724580 >> 172.21.15.28:0/3269724580 conn(0x7fbb300d0430 msgr2=0x7fbb300d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.138+0000 7fbb384af640 1 -- 172.21.15.28:0/3269724580 shutdown_connections 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.138+0000 7fbb384af640 1 -- 172.21.15.28:0/3269724580 wait complete. 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.140+0000 7fbb384af640 1 Processor -- start 2024-04-25T19:13:16.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.140+0000 7fbb384af640 1 -- start start 2024-04-25T19:13:16.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb384af640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18004d40 0x7fbb301f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb384af640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb301dbc90 0x7fbb300b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb384af640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3009f940 0x7fbb3009fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb384af640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb3006e5e0 con 0x7fbb3009f940 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb384af640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbb300b8d60 con 0x7fbb18004d40 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb384af640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3006ea20 con 0x7fbb301dbc90 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb36a4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3009f940 0x7fbb3009fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb36a4c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3009f940 0x7fbb3009fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59560/0 (socket says 172.21.15.28:59560) 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb36a4c640 1 -- 172.21.15.28:0/3683100082 learned_addr learned my addr 172.21.15.28:0/3683100082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb35a4a640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb301dbc90 0x7fbb300b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb3624b640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18004d40 0x7fbb301f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb36a4c640 1 -- 172.21.15.28:0/3683100082 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb301dbc90 msgr2=0x7fbb300b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb36a4c640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb301dbc90 0x7fbb300b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb36a4c640 1 -- 172.21.15.28:0/3683100082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18004d40 msgr2=0x7fbb301f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb36a4c640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18004d40 0x7fbb301f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.141+0000 7fbb36a4c640 1 -- 172.21.15.28:0/3683100082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3009e8a0 con 0x7fbb3009f940 2024-04-25T19:13:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.142+0000 7fbb36a4c640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3009f940 0x7fbb3009fd50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000e230 tx=0x7fbb20009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.142+0000 7fbb2effd640 1 -- 172.21.15.28:0/3683100082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20009c10 con 0x7fbb3009f940 2024-04-25T19:13:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.142+0000 7fbb2effd640 1 -- 172.21.15.28:0/3683100082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb20045430 con 0x7fbb3009f940 2024-04-25T19:13:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.142+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb3009eb80 con 0x7fbb3009f940 2024-04-25T19:13:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.142+0000 7fbb2effd640 1 -- 172.21.15.28:0/3683100082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb20045820 con 0x7fbb3009f940 2024-04-25T19:13:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.142+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb3009f140 con 0x7fbb3009f940 2024-04-25T19:13:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.143+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb300b28c0 con 0x7fbb3009f940 2024-04-25T19:13:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.144+0000 7fbb2effd640 1 -- 172.21.15.28:0/3683100082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbb20005d10 con 0x7fbb3009f940 2024-04-25T19:13:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.144+0000 7fbb2effd640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbb1c0744b0 0x7fbb1c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.144+0000 7fbb3624b640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbb1c0744b0 0x7fbb1c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.144+0000 7fbb2effd640 1 -- 172.21.15.28:0/3683100082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fbb200b63b0 con 0x7fbb3009f940 2024-04-25T19:13:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.145+0000 7fbb3624b640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbb1c0744b0 0x7fbb1c0769c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbb28001eb0 tx=0x7fbb28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:16.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.149+0000 7fbb2effd640 1 -- 172.21.15.28:0/3683100082 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb200801c0 con 0x7fbb3009f940 2024-04-25T19:13:16.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.372+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb30050740 con 0x7fbb3009f940 2024-04-25T19:13:16.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.372+0000 7fbb2effd640 1 -- 172.21.15.28:0/3683100082 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6483 (secure 0 0 0) 0x7fbb200844c0 con 0x7fbb3009f940 2024-04-25T19:13:16.374 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:16.375 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:16.375 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-04-25T19:13:16.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.374+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbb1c0744b0 msgr2=0x7fbb1c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.374+0000 7fbb384af640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbb1c0744b0 0x7fbb1c0769c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbb28001eb0 tx=0x7fbb28009040 comp rx=0 tx=0).stop 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.374+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3009f940 msgr2=0x7fbb3009fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.374+0000 7fbb384af640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3009f940 0x7fbb3009fd50 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbb2000e230 tx=0x7fbb20009750 comp rx=0 tx=0).stop 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.375+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 shutdown_connections 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.375+0000 7fbb384af640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbb1c0744b0 0x7fbb1c0769c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.375+0000 7fbb384af640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb3009f940 0x7fbb3009fd50 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.375+0000 7fbb384af640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbb301dbc90 0x7fbb300b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.375+0000 7fbb384af640 1 --2- 172.21.15.28:0/3683100082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb18004d40 0x7fbb301f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.375+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 >> 172.21.15.28:0/3683100082 conn(0x7fbb300d0430 msgr2=0x7fbb300b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.375+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 shutdown_connections 2024-04-25T19:13:16.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.375+0000 7fbb384af640 1 -- 172.21.15.28:0/3683100082 wait complete. 2024-04-25T19:13:16.388 INFO:tasks.cephfs.test_recovery_pool:{ "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 2, "epoch": 30, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T19:12:44.063277+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 19, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [ 0 ], "flags": 19, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": false, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": {}, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 5, "modified": "2024-04-25T19:12:54.689116+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": {} } }, { "id": 3, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T19:13:03.631282+0000", "damaged": [], "data_pools": [ 6 ], "enabled": true, "epoch": 30, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "recovery_fs", "in": [ 0 ], "info": { "gid_5589": { "addr": "172.21.15.28:6837/1842070529", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6836", "nonce": 1842070529, "type": "v2" }, { "addr": "172.21.15.28:6837", "nonce": 1842070529, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5589, "incarnation": 27, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 7, "modified": "2024-04-25T19:13:13.979864+0000", "qdb_cluster": [ 5589 ], "qdb_leader": 5589, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5589 } } } ], "standbys": [ { "addr": "172.21.15.53:6834/1413023330", "addrs": { "addrvec": [ { "addr": "172.21.15.53:6832", "nonce": 1413023330, "type": "v2" }, { "addr": "172.21.15.53:6834", "nonce": 1413023330, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 21, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5368, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.28:6835/568086068", "addrs": { "addrvec": [ { "addr": "172.21.15.28:6834", "nonce": 568086068, "type": "v2" }, { "addr": "172.21.15.28:6835", "nonce": 568086068, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 20, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5568, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.53:6835/2217485298", "addrs": { "addrvec": [ { "addr": "172.21.15.53:6833", "nonce": 2217485298, "type": "v2" }, { "addr": "172.21.15.53:6835", "nonce": 2217485298, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 21, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5574, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T19:13:16.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T19:13:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.498+0000 7f76db0f2640 1 Processor -- start 2024-04-25T19:13:16.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.498+0000 7f76db0f2640 1 -- start start 2024-04-25T19:13:16.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.499+0000 7f76db0f2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4160120 0x7f76d40b7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.499+0000 7f76db0f2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76d4160530 con 0x7f76d40b7e80 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.499+0000 7f76db0f2640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76d40b8290 con 0x7f76d4160e90 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.499+0000 7f76db0f2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f76d4164580 con 0x7f76d4160120 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d3fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f76d40b7e80 0x7f76d4160780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36984/0 (socket says 172.21.15.28:36984) 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d3fff640 1 -- 172.21.15.28:0/2171409788 learned_addr learned my addr 172.21.15.28:0/2171409788 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030597014 0 0) 0x7f76c8001110 con 0x7f76d40b7e80 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d8e8e640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4160120 0x7f76d40b7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f76bc003520 con 0x7f76d40b7e80 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 31124298 0 0) 0x7f76bc003520 con 0x7f76d40b7e80 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4160120 msgr2=0x7f76d40b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4160120 0x7f76d40b7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 >> v1:172.21.15.53:6789/0 conn(0x7f76d4160e90 legacy=0x7f76d40d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d41614f0 con 0x7f76d40b7e80 2024-04-25T19:13:16.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f76c8001840 con 0x7f76d40b7e80 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 >> v1:172.21.15.28:6789/0 conn(0x7f76d40b7e80 legacy=0x7f76d4160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76bc003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4160120 0x7f76bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc004d40 0x7f76bc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76d4164580 con 0x7f76bc0039e0 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f76bc00baf0 con 0x7f76bc004d40 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f76bc00bcf0 con 0x7f76d4160120 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.500+0000 7f76d8e8e640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76bc003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d3fff640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4160120 0x7f76bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d968f640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc004d40 0x7f76bc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d8e8e640 1 -- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4160120 msgr2=0x7f76bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d8e8e640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4160120 0x7f76bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d8e8e640 1 -- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc004d40 msgr2=0x7f76bc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d8e8e640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc004d40 0x7f76bc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d8e8e640 1 -- 172.21.15.28:0/2171409788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76bc003520 con 0x7f76bc0039e0 2024-04-25T19:13:16.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d968f640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76bc004d40 0x7f76bc005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.501+0000 7f76d8e8e640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76bc003e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009a90 tx=0x7f76bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d240f7f505ad73fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4036070 con 0x7f76bc0039e0 2024-04-25T19:13:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76c4031400 con 0x7f76bc0039e0 2024-04-25T19:13:16.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76d37fe640 1 -- 172.21.15.28:0/2171409788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4009690 con 0x7f76bc0039e0 2024-04-25T19:13:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76db0f2640 1 -- 172.21.15.28:0/2171409788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 msgr2=0x7f76bc003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76db0f2640 1 --2- 172.21.15.28:0/2171409788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76bc003e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009a90 tx=0x7f76bc005830 comp rx=0 tx=0).stop 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76d968f640 1 -- 172.21.15.28:0/2171409788 reap_dead start 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76db0f2640 1 -- 172.21.15.28:0/2171409788 shutdown_connections 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76db0f2640 1 -- 172.21.15.28:0/2171409788 >> 172.21.15.28:0/2171409788 conn(0x7f76d40d2980 msgr2=0x7f76d40d2d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76db0f2640 1 -- 172.21.15.28:0/2171409788 shutdown_connections 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.502+0000 7f76db0f2640 1 -- 172.21.15.28:0/2171409788 wait complete. 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.504+0000 7f76db0f2640 1 Processor -- start 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.504+0000 7f76db0f2640 1 -- start start 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76db0f2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76d415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76db0f2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4159a90 0x7f76d41f0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76db0f2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76d415e060 0x7f76d415e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76db0f2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76d4163f30 con 0x7f76bc0039e0 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76db0f2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f76d41612a0 con 0x7f76d415e060 2024-04-25T19:13:16.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76db0f2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f76d4164210 con 0x7f76d4159a90 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76d415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76d415ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59570/0 (socket says 172.21.15.28:59570) 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 -- 172.21.15.28:0/2555401482 learned_addr learned my addr 172.21.15.28:0/2555401482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d3fff640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4159a90 0x7f76d41f0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d968f640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76d415e060 0x7f76d415e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 -- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4159a90 msgr2=0x7f76d41f0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4159a90 0x7f76d41f0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 -- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76d415e060 msgr2=0x7f76d415e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76d415e060 0x7f76d415e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 -- 172.21.15.28:0/2555401482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d415e9b0 con 0x7f76bc0039e0 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d968f640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76d415e060 0x7f76d415e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d8e8e640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76d415ca40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009af0 tx=0x7f76c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d17fa640 1 -- 172.21.15.28:0/2555401482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4040020 con 0x7f76bc0039e0 2024-04-25T19:13:16.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d17fa640 1 -- 172.21.15.28:0/2555401482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76c403b680 con 0x7f76bc0039e0 2024-04-25T19:13:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d41f71d0 con 0x7f76bc0039e0 2024-04-25T19:13:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.506+0000 7f76d17fa640 1 -- 172.21.15.28:0/2555401482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4036040 con 0x7f76bc0039e0 2024-04-25T19:13:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.507+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d41f66f0 con 0x7f76bc0039e0 2024-04-25T19:13:16.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.507+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d4050740 con 0x7f76bc0039e0 2024-04-25T19:13:16.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.508+0000 7f76d17fa640 1 -- 172.21.15.28:0/2555401482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f76c4043020 con 0x7f76bc0039e0 2024-04-25T19:13:16.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.509+0000 7f76d17fa640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f76c00744e0 0x7f76c00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:16.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.509+0000 7f76d17fa640 1 -- 172.21.15.28:0/2555401482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f76c40b5d10 con 0x7f76bc0039e0 2024-04-25T19:13:16.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.509+0000 7f76d3fff640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f76c00744e0 0x7f76c00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:16.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.509+0000 7f76d3fff640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f76c00744e0 0x7f76c00769f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f76d4161a30 tx=0x7f76c8000ef0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:16.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.514+0000 7f76d17fa640 1 -- 172.21.15.28:0/2555401482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76c404a030 con 0x7f76bc0039e0 2024-04-25T19:13:16.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.741+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f76d415f580 con 0x7f76bc0039e0 2024-04-25T19:13:16.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.741+0000 7f76d17fa640 1 -- 172.21.15.28:0/2555401482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+156 (secure 0 0 0) 0x7f76d415e9b0 con 0x7f76bc0039e0 2024-04-25T19:13:16.743 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T19:13:16.743 INFO:teuthology.orchestra.run.smithi028.stdout:name: recovery_fs, metadata pool: recovery_fs_metadata, data pools: [cephfs_data ] 2024-04-25T19:13:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f76c00744e0 msgr2=0x7f76c00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f76c00744e0 0x7f76c00769f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f76d4161a30 tx=0x7f76c8000ef0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 msgr2=0x7f76d415ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76d415ca40 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f76c4009af0 tx=0x7f76c4004730 comp rx=0 tx=0).stop 2024-04-25T19:13:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 shutdown_connections 2024-04-25T19:13:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f76c00744e0 0x7f76c00769f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76d415e060 0x7f76d415e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f76d4159a90 0x7f76d41f0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 --2- 172.21.15.28:0/2555401482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76bc0039e0 0x7f76d415ca40 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:16.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 >> 172.21.15.28:0/2555401482 conn(0x7f76d40d2980 msgr2=0x7f76d40d6340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:16.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 shutdown_connections 2024-04-25T19:13:16.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16.743+0000 7f76db0f2640 1 -- 172.21.15.28:0/2555401482 wait complete. 2024-04-25T19:13:16.756 INFO:tasks.cephfs.mount:Mounting Ceph FS recovery_fs; just confirmed its presence on cluster 2024-04-25T19:13:16.756 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T19:13:16.756 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-25T19:13:16.756 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T19:13:16.756 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T19:13:16.756 INFO:tasks.cephfs.mount:self.cephfs_name = recovery_fs 2024-04-25T19:13:16.756 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T19:13:16.756 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T19:13:16.757 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T19:13:16.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:16.757 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-25T19:13:16.772 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T19:13:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T19:13:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42663sec preferred_lft 42663sec 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether b6:8b:9a:39:76:98 brd ff:ff:ff:ff:ff:ff 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::d015:3fff:fe07:1fb/64 scope link 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T19:13:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether b6:8b:9a:39:76:98 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T19:13:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::b48b:9aff:fe39:7698/64 scope link 2024-04-25T19:13:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:13:16.775 INFO:teuthology.orchestra.run.smithi028.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T19:13:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether e6:ae:c1:be:ae:c4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T19:13:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::e4ae:c1ff:febe:aec4/64 scope link 2024-04-25T19:13:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-25T19:13:16.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:16.775 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T19:13:16.835 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-25T19:13:16.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:16.837 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-25T19:13:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-25T19:13:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T19:13:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T19:13:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T19:13:16.895 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T19:13:16.895 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T19:13:16.895 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-25T19:13:16.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T19:13:16.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T19:13:16.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T19:13:16.896 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-25T19:13:16.924 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-04-25T19:13:16.922+0000 7f95c9b60640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T19:13:16.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T19:13:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:13:17.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:17.011 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:13:17.026 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T19:13:17.027 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T19:13:17.027 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T19:13:17.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:17.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-25T19:13:17.091 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:13:17.091 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T19:13:17.146 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T19:13:17.146 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T19:13:17.146 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T19:13:17.201 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.recovery_fs=/ 2024-04-25T19:13:17.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:17.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.recovery_fs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:13:17.294 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:13:17.295 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:13:17.304 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T19:13:17.304 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T19:13:17.304 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.53:3300,172.21.15.53:3301" 2024-04-25T19:13:17.304 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@5d8007d4-24a5-41be-aa4d-cd64fedac62a.recovery_fs=/ 2024-04-25T19:13:17.304 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.53:3300/172.21.15.53:3301" will pass to kernel 2024-04-25T19:13:17.305 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T19:13:17.305 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-04-25T19:13:17.305 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T19:13:17.305 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T19:13:17.305 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T19:13:17.306 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T19:13:17.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:17.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T19:13:17.336 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T19:13:17.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T19:13:17.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:13:17.437 INFO:teuthology.orchestra.run.smithi028.stdout:client5156 2024-04-25T19:13:17.437 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:13:17.437 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:17.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:17.453 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir) 2024-04-25T19:13:17.512 INFO:teuthology.orchestra.run.smithi028.stdout:sixmegs 2024-04-25T19:13:17.512 DEBUG:teuthology.orchestra.run.smithi028:> stdin-killer -- python3 -c ' 2024-04-25T19:13:17.512 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-04-25T19:13:17.512 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-04-25T19:13:17.512 DEBUG:teuthology.orchestra.run.smithi028:> import json 2024-04-25T19:13:17.512 DEBUG:teuthology.orchestra.run.smithi028:> import sys 2024-04-25T19:13:17.512 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T19:13:17.512 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-04-25T19:13:17.512 DEBUG:teuthology.orchestra.run.smithi028:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-04-25T19:13:17.513 DEBUG:teuthology.orchestra.run.smithi028:> except OSError as e: 2024-04-25T19:13:17.513 DEBUG:teuthology.orchestra.run.smithi028:> sys.exit(e.errno) 2024-04-25T19:13:17.513 DEBUG:teuthology.orchestra.run.smithi028:> 2024-04-25T19:13:17.513 DEBUG:teuthology.orchestra.run.smithi028:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-04-25T19:13:17.513 DEBUG:teuthology.orchestra.run.smithi028:> print(json.dumps( 2024-04-25T19:13:17.513 DEBUG:teuthology.orchestra.run.smithi028:> dict([(a, getattr(s, a)) for a in attrs]), 2024-04-25T19:13:17.513 DEBUG:teuthology.orchestra.run.smithi028:> indent=2)) 2024-04-25T19:13:17.513 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-04-25T19:13:17.625 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mode": 33088, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ino": 1099511627777, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_dev": 42, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_nlink": 1, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_uid": 0, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_gid": 0, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_size": 6291456, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_atime": 1714072374.0, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mtime": 1714072374.0, 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ctime": 1714072374.0 2024-04-25T19:13:17.626 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:17.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T19:13:17.636 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T19:13:17.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T19:13:17.718 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T19:13:17.766 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T19:13:17.766 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T19:13:17.767 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:13:17.767 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:17.782 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:13:17.782 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:17.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.987+0000 7f264a564640 1 Processor -- start 2024-04-25T19:13:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.987+0000 7f264a564640 1 -- start start 2024-04-25T19:13:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f264a564640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f264409cb20 0x7f26440b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f264a564640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2644163850 0x7f26440ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:17.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f264a564640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26440ce550 0x7f26440d2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f264a564640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f26440d2620 con 0x7f26440ce550 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f264a564640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f26440d1d30 con 0x7f264409cb20 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f264a564640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f264406ebb0 con 0x7f2644163850 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2648b01640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26440ce550 0x7f26440d2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2648b01640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26440ce550 0x7f26440d2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59584/0 (socket says 172.21.15.28:59584) 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2648b01640 1 -- 172.21.15.28:0/3428861448 learned_addr learned my addr 172.21.15.28:0/3428861448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f26437fe640 1 --2- 172.21.15.28:0/3428861448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2644163850 0x7f26440ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2643fff640 1 --2- 172.21.15.28:0/3428861448 >> v2:172.21.15.53:3300/0 conn(0x7f264409cb20 0x7f26440b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2648b01640 1 -- 172.21.15.28:0/3428861448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2644163850 msgr2=0x7f26440ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2648b01640 1 --2- 172.21.15.28:0/3428861448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2644163850 0x7f26440ce010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2648b01640 1 -- 172.21.15.28:0/3428861448 >> v2:172.21.15.53:3300/0 conn(0x7f264409cb20 msgr2=0x7f26440b91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:17.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2648b01640 1 --2- 172.21.15.28:0/3428861448 >> v2:172.21.15.53:3300/0 conn(0x7f264409cb20 0x7f26440b91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2648b01640 1 -- 172.21.15.28:0/3428861448 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f264406e5f0 con 0x7f26440ce550 2024-04-25T19:13:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.989+0000 7f2643fff640 1 --2- 172.21.15.28:0/3428861448 >> v2:172.21.15.53:3300/0 conn(0x7f264409cb20 0x7f26440b91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:17.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.990+0000 7f2648b01640 1 --2- 172.21.15.28:0/3428861448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26440ce550 0x7f26440d2ab0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f264408cde0 tx=0x7f263c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f524d1d634093919 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.990+0000 7f2642ffd640 1 -- 172.21.15.28:0/3428861448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c036070 con 0x7f26440ce550 2024-04-25T19:13:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.990+0000 7f2642ffd640 1 -- 172.21.15.28:0/3428861448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f263c031600 con 0x7f26440ce550 2024-04-25T19:13:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.990+0000 7f2642ffd640 1 -- 172.21.15.28:0/3428861448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f263c0319d0 con 0x7f26440ce550 2024-04-25T19:13:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 -- 172.21.15.28:0/3428861448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26440ce550 msgr2=0x7f26440d2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 --2- 172.21.15.28:0/3428861448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26440ce550 0x7f26440d2ab0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f264408cde0 tx=0x7f263c02a6a0 comp rx=0 tx=0).stop 2024-04-25T19:13:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 -- 172.21.15.28:0/3428861448 shutdown_connections 2024-04-25T19:13:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 --2- 172.21.15.28:0/3428861448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26440ce550 0x7f26440d2ab0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:17.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 --2- 172.21.15.28:0/3428861448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2644163850 0x7f26440ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 --2- 172.21.15.28:0/3428861448 >> v2:172.21.15.53:3300/0 conn(0x7f264409cb20 0x7f26440b91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 -- 172.21.15.28:0/3428861448 >> 172.21.15.28:0/3428861448 conn(0x7f26440b54b0 msgr2=0x7f26440b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 -- 172.21.15.28:0/3428861448 shutdown_connections 2024-04-25T19:13:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.991+0000 7f264a564640 1 -- 172.21.15.28:0/3428861448 wait complete. 2024-04-25T19:13:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.993+0000 7f264a564640 1 Processor -- start 2024-04-25T19:13:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.993+0000 7f264a564640 1 -- start start 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f264a564640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f264409cb20 0x7f2644159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f264a564640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26440ce550 0x7f26440b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f264a564640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2644163850 0x7f2644161570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f264a564640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26440d22b0 con 0x7f264409cb20 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f264a564640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f26440d1fd0 con 0x7f2644163850 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f264a564640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2644163c60 con 0x7f26440ce550 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f2643fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f264409cb20 0x7f2644159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f2643fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f264409cb20 0x7f2644159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59588/0 (socket says 172.21.15.28:59588) 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.994+0000 7f2643fff640 1 -- 172.21.15.28:0/3579764030 learned_addr learned my addr 172.21.15.28:0/3579764030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f2648b01640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2644163850 0x7f2644161570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:17.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f26437fe640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26440ce550 0x7f26440b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f2643fff640 1 -- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26440ce550 msgr2=0x7f26440b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f2643fff640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26440ce550 0x7f26440b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f2643fff640 1 -- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2644163850 msgr2=0x7f2644161570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f2643fff640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2644163850 0x7f2644161570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f2643fff640 1 -- 172.21.15.28:0/3579764030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26440b2640 con 0x7f264409cb20 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f2648b01640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2644163850 0x7f2644161570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.995+0000 7f2643fff640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f264409cb20 0x7f2644159300 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f262c00c820 tx=0x7f262c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.996+0000 7f2640ff9640 1 -- 172.21.15.28:0/3579764030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c015070 con 0x7f264409cb20 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.996+0000 7f2640ff9640 1 -- 172.21.15.28:0/3579764030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262c007830 con 0x7f264409cb20 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.996+0000 7f2640ff9640 1 -- 172.21.15.28:0/3579764030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262c007c20 con 0x7f264409cb20 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.996+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2644159ac0 con 0x7f264409cb20 2024-04-25T19:13:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.996+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f264415a080 con 0x7f264409cb20 2024-04-25T19:13:18.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.997+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2644050740 con 0x7f264409cb20 2024-04-25T19:13:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.997+0000 7f2640ff9640 1 -- 172.21.15.28:0/3579764030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f262c018070 con 0x7f264409cb20 2024-04-25T19:13:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.998+0000 7f2640ff9640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26240744e0 0x7f26240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.998+0000 7f2640ff9640 1 -- 172.21.15.28:0/3579764030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f262c095330 con 0x7f264409cb20 2024-04-25T19:13:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.998+0000 7f26437fe640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26240744e0 0x7f26240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:17.998+0000 7f26437fe640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26240744e0 0x7f26240769f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2634006fd0 tx=0x7f2634008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.003+0000 7f2640ff9640 1 -- 172.21.15.28:0/3579764030 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f262c05f790 con 0x7f264409cb20 2024-04-25T19:13:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.190+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f26440cff30 con 0x7f264409cb20 2024-04-25T19:13:18.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.192+0000 7f2640ff9640 1 -- 172.21.15.28:0/3579764030 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+18884 (secure 0 0 0) 0x7f262c05f090 con 0x7f264409cb20 2024-04-25T19:13:18.194 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:18.194 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:10.963059+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26240744e0 msgr2=0x7f26240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26240744e0 0x7f26240769f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2634006fd0 tx=0x7f2634008040 comp rx=0 tx=0).stop 2024-04-25T19:13:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f264409cb20 msgr2=0x7f2644159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f264409cb20 0x7f2644159300 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f262c00c820 tx=0x7f262c00cce0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 shutdown_connections 2024-04-25T19:13:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26240744e0 0x7f26240769f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2644163850 0x7f2644161570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26440ce550 0x7f26440b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 --2- 172.21.15.28:0/3579764030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f264409cb20 0x7f2644159300 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 >> 172.21.15.28:0/3579764030 conn(0x7f26440b54b0 msgr2=0x7f2644072da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 shutdown_connections 2024-04-25T19:13:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.194+0000 7f264a564640 1 -- 172.21.15.28:0/3579764030 wait complete. 2024-04-25T19:13:18.207 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:12:41.809696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:12:43.014197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-04-25T19:13:02.679639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:13:18.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:13:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.320+0000 7f7d88ce7640 1 Processor -- start 2024-04-25T19:13:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.320+0000 7f7d88ce7640 1 -- start start 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d88ce7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d840b3d40 0x7f7d840b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d88ce7640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d840d2650 con 0x7f7d840b9090 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d88ce7640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d840d2930 con 0x7f7d840b89d0 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d88ce7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7d840d2c10 con 0x7f7d840b3d40 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d8259c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7d840b9090 0x7f7d840cf3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37000/0 (socket says 172.21.15.28:37000) 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d8259c640 1 -- 172.21.15.28:0/3353315114 learned_addr learned my addr 172.21.15.28:0/3353315114 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d82d9d640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d840b3d40 0x7f7d840b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3164184257 0 0) 0x7f7d840d2650 con 0x7f7d840b9090 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d70003520 con 0x7f7d840b9090 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3417425646 0 0) 0x7f7d70003520 con 0x7f7d840b9090 2024-04-25T19:13:18.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d840b3d40 msgr2=0x7f7d840b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d840b3d40 0x7f7d840b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 >> v1:172.21.15.53:6789/0 conn(0x7f7d840b89d0 legacy=0x7f7d840cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8409cc90 con 0x7f7d840b9090 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f7d74001690 con 0x7f7d840b9090 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.322+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 >> v1:172.21.15.28:6789/0 conn(0x7f7d840b9090 legacy=0x7f7d840cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d81d9b640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d70003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d81d9b640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d70004800 0x7f7d70004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d81d9b640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d840b3d40 0x7f7d700051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d840d2c10 con 0x7f7d700039e0 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7000bad0 con 0x7f7d70004800 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7000bcd0 con 0x7f7d840b3d40 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d8259c640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d70003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d7bfff640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d70004800 0x7f7d70004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d82d9d640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d840b3d40 0x7f7d700051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d8259c640 1 -- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d840b3d40 msgr2=0x7f7d700051b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d8259c640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d840b3d40 0x7f7d700051b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d8259c640 1 -- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d70004800 msgr2=0x7f7d70004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d8259c640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d70004800 0x7f7d70004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d8259c640 1 -- 172.21.15.28:0/3353315114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d70003520 con 0x7f7d700039e0 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.323+0000 7f7d7bfff640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d70004800 0x7f7d70004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.324+0000 7f7d8259c640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d70003e10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7d840b5490 tx=0x7f7d70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d1b7f249c042636 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.324+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74007ca0 con 0x7f7d700039e0 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.324+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d74005af0 con 0x7f7d700039e0 2024-04-25T19:13:18.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.324+0000 7f7d81d9b640 1 -- 172.21.15.28:0/3353315114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74007070 con 0x7f7d700039e0 2024-04-25T19:13:18.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.324+0000 7f7d88ce7640 1 -- 172.21.15.28:0/3353315114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 msgr2=0x7f7d70003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.324+0000 7f7d88ce7640 1 --2- 172.21.15.28:0/3353315114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d70003e10 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7d840b5490 tx=0x7f7d70005830 comp rx=0 tx=0).stop 2024-04-25T19:13:18.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.324+0000 7f7d82d9d640 1 -- 172.21.15.28:0/3353315114 reap_dead start 2024-04-25T19:13:18.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.325+0000 7f7d88ce7640 1 -- 172.21.15.28:0/3353315114 shutdown_connections 2024-04-25T19:13:18.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.325+0000 7f7d88ce7640 1 -- 172.21.15.28:0/3353315114 >> 172.21.15.28:0/3353315114 conn(0x7f7d840d0450 msgr2=0x7f7d840d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:18.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.325+0000 7f7d88ce7640 1 -- 172.21.15.28:0/3353315114 shutdown_connections 2024-04-25T19:13:18.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.325+0000 7f7d88ce7640 1 -- 172.21.15.28:0/3353315114 wait complete. 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.327+0000 7f7d88ce7640 1 Processor -- start 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.327+0000 7f7d88ce7640 1 -- start start 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d88ce7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d841d9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d88ce7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d841f75a0 0x7f7d841f79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d88ce7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d8409f3c0 0x7f7d8409f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d88ce7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d840cfaf0 con 0x7f7d700039e0 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d88ce7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7d8409d000 con 0x7f7d841f75a0 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d88ce7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8408cd90 con 0x7f7d8409f3c0 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d8259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d841d9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d8259c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d841d9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59606/0 (socket says 172.21.15.28:59606) 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d8259c640 1 -- 172.21.15.28:0/42585456 learned_addr learned my addr 172.21.15.28:0/42585456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d7bfff640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d841f75a0 0x7f7d841f79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d82d9d640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d8409f3c0 0x7f7d8409f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d8259c640 1 -- 172.21.15.28:0/42585456 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d8409f3c0 msgr2=0x7f7d8409f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d8259c640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d8409f3c0 0x7f7d8409f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d8259c640 1 -- 172.21.15.28:0/42585456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d841f75a0 msgr2=0x7f7d841f79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d8259c640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d841f75a0 0x7f7d841f79b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.328+0000 7f7d8259c640 1 -- 172.21.15.28:0/42585456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d841f7ef0 con 0x7f7d700039e0 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.329+0000 7f7d8259c640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d841d9ae0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7d74007fd0 tx=0x7f7d74001990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.329+0000 7f7d7affd640 1 -- 172.21.15.28:0/42585456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74042070 con 0x7f7d700039e0 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.329+0000 7f7d7affd640 1 -- 172.21.15.28:0/42585456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d740047e0 con 0x7f7d700039e0 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.329+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8409e600 con 0x7f7d700039e0 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.329+0000 7f7d7affd640 1 -- 172.21.15.28:0/42585456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74037040 con 0x7f7d700039e0 2024-04-25T19:13:18.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.329+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d8409ebc0 con 0x7f7d700039e0 2024-04-25T19:13:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.330+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d84050740 con 0x7f7d700039e0 2024-04-25T19:13:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.331+0000 7f7d7affd640 1 -- 172.21.15.28:0/42585456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7d74045070 con 0x7f7d700039e0 2024-04-25T19:13:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.331+0000 7f7d7affd640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7d640744e0 0x7f7d640769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.331+0000 7f7d7affd640 1 -- 172.21.15.28:0/42585456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f7d740b4dc0 con 0x7f7d700039e0 2024-04-25T19:13:18.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.331+0000 7f7d7bfff640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7d640744e0 0x7f7d640769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.331+0000 7f7d7bfff640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7d640744e0 0x7f7d640769f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c007fd0 tx=0x7f7d6c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.336+0000 7f7d7affd640 1 -- 172.21.15.28:0/42585456 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d841f7ef0 con 0x7f7d700039e0 2024-04-25T19:13:18.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.520+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7d840cfdf0 con 0x7f7d700039e0 2024-04-25T19:13:18.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.520+0000 7f7d7affd640 1 -- 172.21.15.28:0/42585456 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f7d74087210 con 0x7f7d700039e0 2024-04-25T19:13:18.526 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7d640744e0 msgr2=0x7f7d640769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7d640744e0 0x7f7d640769f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c007fd0 tx=0x7f7d6c009040 comp rx=0 tx=0).stop 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 msgr2=0x7f7d841d9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d841d9ae0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7d74007fd0 tx=0x7f7d74001990 comp rx=0 tx=0).stop 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 shutdown_connections 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7d640744e0 0x7f7d640769f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7d8409f3c0 0x7f7d8409f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d841f75a0 0x7f7d841f79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 --2- 172.21.15.28:0/42585456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d700039e0 0x7f7d841d9ae0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 >> 172.21.15.28:0/42585456 conn(0x7f7d840d0450 msgr2=0x7f7d840cea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 shutdown_connections 2024-04-25T19:13:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.522+0000 7f7d88ce7640 1 -- 172.21.15.28:0/42585456 wait complete. 2024-04-25T19:13:18.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:13:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.649+0000 7f5982e27640 1 Processor -- start 2024-04-25T19:13:18.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.649+0000 7f5982e27640 1 -- start start 2024-04-25T19:13:18.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.650+0000 7f5982e27640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f597c164990 0x7f597c15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.650+0000 7f5982e27640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0b7660 0x7f597c0b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.650+0000 7f5982e27640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f597c161210 con 0x7f597c1606f0 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.650+0000 7f5982e27640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f597c0b7fb0 con 0x7f597c164990 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.650+0000 7f5982e27640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f597c1614a0 con 0x7f597c0b7660 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.650+0000 7f59813c4640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f597c1606f0 0x7f597c160b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37004/0 (socket says 172.21.15.28:37004) 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.650+0000 7f59813c4640 1 -- 172.21.15.28:0/1350262023 learned_addr learned my addr 172.21.15.28:0/1350262023 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f5980bc3640 1 --2- 172.21.15.28:0/1350262023 >> v2:172.21.15.53:3300/0 conn(0x7f597c164990 0x7f597c15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597bfff640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0b7660 0x7f597c0b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851901585 0 0) 0x7f597c161210 con 0x7f597c1606f0 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5968003520 con 0x7f597c1606f0 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2635424661 0 0) 0x7f5968003520 con 0x7f597c1606f0 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0b7660 msgr2=0x7f597c0b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0b7660 0x7f597c0b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 >> v2:172.21.15.53:3300/0 conn(0x7f597c164990 msgr2=0x7f597c15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:18.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 --2- 172.21.15.28:0/1350262023 >> v2:172.21.15.53:3300/0 conn(0x7f597c164990 0x7f597c15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c163eb0 con 0x7f597c1606f0 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f59740019f0 con 0x7f597c1606f0 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 >> v1:172.21.15.28:6789/0 conn(0x7f597c1606f0 legacy=0x7f597c160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59680039b0 0x7f5968003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0b7660 0x7f5968004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5968004d40 0x7f5968005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f597c0b7fb0 con 0x7f59680039b0 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f597c1614a0 con 0x7f5968004d40 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f596800b9c0 con 0x7f597c0b7660 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f5980bc3640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59680039b0 0x7f5968003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f597bfff640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0b7660 0x7f5968004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f59813c4640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5968004d40 0x7f5968005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f5980bc3640 1 -- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0b7660 msgr2=0x7f5968004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f5980bc3640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0b7660 0x7f5968004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f5980bc3640 1 -- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5968004d40 msgr2=0x7f5968005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.651+0000 7f5980bc3640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5968004d40 0x7f5968005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.652+0000 7f5980bc3640 1 -- 172.21.15.28:0/1350262023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5968003520 con 0x7f59680039b0 2024-04-25T19:13:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.652+0000 7f5980bc3640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59680039b0 0x7f5968003e20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5964009a90 tx=0x7f5968005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b7ddc427a1862a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.652+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964036070 con 0x7f59680039b0 2024-04-25T19:13:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.652+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5964031400 con 0x7f59680039b0 2024-04-25T19:13:18.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.652+0000 7f597b7fe640 1 -- 172.21.15.28:0/1350262023 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964009690 con 0x7f59680039b0 2024-04-25T19:13:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.653+0000 7f5982e27640 1 -- 172.21.15.28:0/1350262023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59680039b0 msgr2=0x7f5968003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.653+0000 7f5982e27640 1 --2- 172.21.15.28:0/1350262023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59680039b0 0x7f5968003e20 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5964009a90 tx=0x7f5968005830 comp rx=0 tx=0).stop 2024-04-25T19:13:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.653+0000 7f59813c4640 1 -- 172.21.15.28:0/1350262023 reap_dead start 2024-04-25T19:13:18.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.653+0000 7f5982e27640 1 -- 172.21.15.28:0/1350262023 shutdown_connections 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.653+0000 7f5982e27640 1 -- 172.21.15.28:0/1350262023 >> 172.21.15.28:0/1350262023 conn(0x7f597c0d2840 msgr2=0x7f597c0d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.653+0000 7f5982e27640 1 -- 172.21.15.28:0/1350262023 shutdown_connections 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.653+0000 7f5982e27640 1 -- 172.21.15.28:0/1350262023 wait complete. 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.655+0000 7f5982e27640 1 Processor -- start 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.655+0000 7f5982e27640 1 -- start start 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.656+0000 7f5982e27640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680039b0 0x7f597c158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.656+0000 7f5982e27640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c1f6350 0x7f597c0ca410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.656+0000 7f5982e27640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0ca950 0x7f597c0d6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.656+0000 7f5982e27640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f597c0b7a70 con 0x7f597c1f6350 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.656+0000 7f5982e27640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f597c164470 con 0x7f59680039b0 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.656+0000 7f5982e27640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f597c0b7bf0 con 0x7f597c0ca950 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.656+0000 7f597bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c1f6350 0x7f597c0ca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f597bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c1f6350 0x7f597c0ca410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59614/0 (socket says 172.21.15.28:59614) 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f597bfff640 1 -- 172.21.15.28:0/1342121077 learned_addr learned my addr 172.21.15.28:0/1342121077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:18.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f59813c4640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0ca950 0x7f597c0d6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f597bfff640 1 -- 172.21.15.28:0/1342121077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0ca950 msgr2=0x7f597c0d6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f5980bc3640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680039b0 0x7f597c158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f597bfff640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0ca950 0x7f597c0d6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f597bfff640 1 -- 172.21.15.28:0/1342121077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680039b0 msgr2=0x7f597c158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f597bfff640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680039b0 0x7f597c158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f597bfff640 1 -- 172.21.15.28:0/1342121077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c0d5460 con 0x7f597c1f6350 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f597bfff640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c1f6350 0x7f597c0ca410 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f596c0119a0 tx=0x7f596c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f59797fa640 1 -- 172.21.15.28:0/1342121077 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c00fa90 con 0x7f597c1f6350 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f59797fa640 1 -- 172.21.15.28:0/1342121077 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f596c00fc30 con 0x7f597c1f6350 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f59797fa640 1 -- 172.21.15.28:0/1342121077 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596c014440 con 0x7f597c1f6350 2024-04-25T19:13:18.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597c0d5740 con 0x7f597c1f6350 2024-04-25T19:13:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.657+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f597c1f6a70 con 0x7f597c1f6350 2024-04-25T19:13:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.658+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f597c050740 con 0x7f597c1f6350 2024-04-25T19:13:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.659+0000 7f59797fa640 1 -- 172.21.15.28:0/1342121077 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f596c012040 con 0x7f597c1f6350 2024-04-25T19:13:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.659+0000 7f59797fa640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f595c0744e0 0x7f595c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.659+0000 7f59797fa640 1 -- 172.21.15.28:0/1342121077 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f596c095bf0 con 0x7f597c1f6350 2024-04-25T19:13:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.659+0000 7f5980bc3640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f595c0744e0 0x7f595c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.660+0000 7f5980bc3640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f595c0744e0 0x7f595c0769f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5964033000 tx=0x7f5964008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.664+0000 7f59797fa640 1 -- 172.21.15.28:0/1342121077 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f596c019030 con 0x7f597c1f6350 2024-04-25T19:13:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.849+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f597c0d4000 con 0x7f597c1f6350 2024-04-25T19:13:18.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.850+0000 7f59797fa640 1 -- 172.21.15.28:0/1342121077 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f596c05ff20 con 0x7f597c1f6350 2024-04-25T19:13:18.852 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:13:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f595c0744e0 msgr2=0x7f595c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f595c0744e0 0x7f595c0769f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5964033000 tx=0x7f5964008030 comp rx=0 tx=0).stop 2024-04-25T19:13:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c1f6350 msgr2=0x7f597c0ca410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c1f6350 0x7f597c0ca410 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f596c0119a0 tx=0x7f596c011e60 comp rx=0 tx=0).stop 2024-04-25T19:13:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 shutdown_connections 2024-04-25T19:13:18.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f595c0744e0 0x7f595c0769f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f597c0ca950 0x7f597c0d6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f597c1f6350 0x7f597c0ca410 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 --2- 172.21.15.28:0/1342121077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680039b0 0x7f597c158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 >> 172.21.15.28:0/1342121077 conn(0x7f597c0d2840 msgr2=0x7f597c1616e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 shutdown_connections 2024-04-25T19:13:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.852+0000 7f5982e27640 1 -- 172.21.15.28:0/1342121077 wait complete. 2024-04-25T19:13:18.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:13:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.978+0000 7f4722267640 1 Processor -- start 2024-04-25T19:13:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.978+0000 7f4722267640 1 -- start start 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.979+0000 7f4722267640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f471c1696c0 0x7f471c16bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.979+0000 7f4722267640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f471c0b92a0 con 0x7f471c0d2d10 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.979+0000 7f4722267640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f471c09cea0 con 0x7f471c0b3d00 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.979+0000 7f4722267640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f471c0d28d0 con 0x7f471c1696c0 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.979+0000 7f471bfff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f471c0d2d10 0x7f471c0d21c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37014/0 (socket says 172.21.15.28:37014) 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.979+0000 7f471bfff640 1 -- 172.21.15.28:0/840698631 learned_addr learned my addr 172.21.15.28:0/840698631 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.979+0000 7f4720804640 1 --2- 172.21.15.28:0/840698631 >> v2:172.21.15.28:3300/0 conn(0x7f471c1696c0 0x7f471c16bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.980+0000 7f471affd640 1 -- 172.21.15.28:0/840698631 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 695074213 0 0) 0x7f471c0b92a0 con 0x7f471c0d2d10 2024-04-25T19:13:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.980+0000 7f471affd640 1 -- 172.21.15.28:0/840698631 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4704003520 con 0x7f471c0d2d10 2024-04-25T19:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.980+0000 7f4720804640 1 -- 172.21.15.28:0/840698631 >> v1:172.21.15.53:6789/0 conn(0x7f471c0b3d00 legacy=0x7f471c0b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.980+0000 7f4720804640 1 -- 172.21.15.28:0/840698631 >> v1:172.21.15.28:6789/0 conn(0x7f471c0d2d10 legacy=0x7f471c0d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.980+0000 7f4720804640 1 -- 172.21.15.28:0/840698631 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f471c06e8e0 con 0x7f471c1696c0 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f4720804640 1 --2- 172.21.15.28:0/840698631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c1696c0 0x7f471c16bae0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4714009d90 tx=0x7f471402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffc3e64b2d8cf31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f471affd640 1 -- 172.21.15.28:0/840698631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714007740 con 0x7f471c1696c0 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f471affd640 1 -- 172.21.15.28:0/840698631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47140078e0 con 0x7f471c1696c0 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f471affd640 1 -- 172.21.15.28:0/840698631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4714007c90 con 0x7f471c1696c0 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f4722267640 1 -- 172.21.15.28:0/840698631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c1696c0 msgr2=0x7f471c16bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f4722267640 1 --2- 172.21.15.28:0/840698631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c1696c0 0x7f471c16bae0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4714009d90 tx=0x7f471402a660 comp rx=0 tx=0).stop 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f4722267640 1 -- 172.21.15.28:0/840698631 shutdown_connections 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f4722267640 1 --2- 172.21.15.28:0/840698631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c1696c0 0x7f471c16bae0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.981+0000 7f4722267640 1 -- 172.21.15.28:0/840698631 >> 172.21.15.28:0/840698631 conn(0x7f471c0d0450 msgr2=0x7f471c0d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:18.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.982+0000 7f4722267640 1 -- 172.21.15.28:0/840698631 shutdown_connections 2024-04-25T19:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.982+0000 7f4722267640 1 -- 172.21.15.28:0/840698631 wait complete. 2024-04-25T19:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.983+0000 7f4722267640 1 Processor -- start 2024-04-25T19:13:18.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.984+0000 7f4722267640 1 -- start start 2024-04-25T19:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f4722267640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c0b3d00 0x7f471c162bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f4722267640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c0d2d10 0x7f471c163100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f4722267640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c1696c0 0x7f471c1617c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f4722267640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f471c06ebc0 con 0x7f471c0b3d00 2024-04-25T19:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f4722267640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f471c06e690 con 0x7f471c0d2d10 2024-04-25T19:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f4722267640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f471c0d3120 con 0x7f471c1696c0 2024-04-25T19:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c0b3d00 0x7f471c162bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471bfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c0b3d00 0x7f471c162bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59644/0 (socket says 172.21.15.28:59644) 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471bfff640 1 -- 172.21.15.28:0/408178726 learned_addr learned my addr 172.21.15.28:0/408178726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f4720804640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c1696c0 0x7f471c1617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471b7fe640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c0d2d10 0x7f471c163100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471bfff640 1 -- 172.21.15.28:0/408178726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c1696c0 msgr2=0x7f471c1617c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471bfff640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c1696c0 0x7f471c1617c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471bfff640 1 -- 172.21.15.28:0/408178726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c0d2d10 msgr2=0x7f471c163100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471bfff640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c0d2d10 0x7f471c163100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.985+0000 7f471bfff640 1 -- 172.21.15.28:0/408178726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f471c0b1c50 con 0x7f471c0b3d00 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.986+0000 7f471bfff640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c0b3d00 0x7f471c162bc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f470c00c3e0 tx=0x7f470c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.986+0000 7f4718ff9640 1 -- 172.21.15.28:0/408178726 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c017070 con 0x7f471c0b3d00 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.986+0000 7f4718ff9640 1 -- 172.21.15.28:0/408178726 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f470c005ce0 con 0x7f471c0b3d00 2024-04-25T19:13:18.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.986+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f471c0b1f30 con 0x7f471c0b3d00 2024-04-25T19:13:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.986+0000 7f4718ff9640 1 -- 172.21.15.28:0/408178726 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f470c0072d0 con 0x7f471c0b3d00 2024-04-25T19:13:18.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.986+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f471c0b24f0 con 0x7f471c0b3d00 2024-04-25T19:13:18.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.987+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f471c0d1750 con 0x7f471c0b3d00 2024-04-25T19:13:18.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.988+0000 7f4718ff9640 1 -- 172.21.15.28:0/408178726 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f470c01a070 con 0x7f471c0b3d00 2024-04-25T19:13:18.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.988+0000 7f4718ff9640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47080744b0 0x7f47080769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:18.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.988+0000 7f4718ff9640 1 -- 172.21.15.28:0/408178726 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f470c094950 con 0x7f471c0b3d00 2024-04-25T19:13:18.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.988+0000 7f471b7fe640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47080744b0 0x7f47080769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:18.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.988+0000 7f471b7fe640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47080744b0 0x7f47080769c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f471c0b7800 tx=0x7f4710009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:18.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:18.993+0000 7f4718ff9640 1 -- 172.21.15.28:0/408178726 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f471c0d1750 con 0x7f471c0b3d00 2024-04-25T19:13:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.176+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f471c050740 con 0x7f471c0b3d00 2024-04-25T19:13:19.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.177+0000 7f4718ff9640 1 -- 172.21.15.28:0/408178726 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7f470c05e6d0 con 0x7f471c0b3d00 2024-04-25T19:13:19.179 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47080744b0 msgr2=0x7f47080769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47080744b0 0x7f47080769c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f471c0b7800 tx=0x7f4710009040 comp rx=0 tx=0).stop 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c0b3d00 msgr2=0x7f471c162bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c0b3d00 0x7f471c162bc0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f470c00c3e0 tx=0x7f470c00c8a0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 shutdown_connections 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f47080744b0 0x7f47080769c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f471c1696c0 0x7f471c1617c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f471c0d2d10 0x7f471c163100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 --2- 172.21.15.28:0/408178726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f471c0b3d00 0x7f471c162bc0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.178+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 >> 172.21.15.28:0/408178726 conn(0x7f471c0d0450 msgr2=0x7f471c0d21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:19.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.179+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 shutdown_connections 2024-04-25T19:13:19.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.179+0000 7f4722267640 1 -- 172.21.15.28:0/408178726 wait complete. 2024-04-25T19:13:19.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-04-25T19:13:19.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.305+0000 7f35ce9a8640 1 Processor -- start 2024-04-25T19:13:19.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.305+0000 7f35ce9a8640 1 -- start start 2024-04-25T19:13:19.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35ce9a8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35c80affc0 0x7f35c8167d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35ce9a8640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c80b5730 con 0x7f35c80d0ee0 2024-04-25T19:13:19.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35ce9a8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35c806f5b0 con 0x7f35c815aaa0 2024-04-25T19:13:19.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35ce9a8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f35c80af290 con 0x7f35c80affc0 2024-04-25T19:13:19.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35c77fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f35c80d0ee0 0x7f35c80b5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37022/0 (socket says 172.21.15.28:37022) 2024-04-25T19:13:19.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35ccf45640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f35c80affc0 0x7f35c8167d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35c77fe640 1 -- 172.21.15.28:0/4045721198 learned_addr learned my addr 172.21.15.28:0/4045721198 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:19.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35c6ffd640 1 -- 172.21.15.28:0/4045721198 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2951161662 0 0) 0x7f35b0001110 con 0x7f35c80d0ee0 2024-04-25T19:13:19.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35c6ffd640 1 -- 172.21.15.28:0/4045721198 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35b4003520 con 0x7f35c80d0ee0 2024-04-25T19:13:19.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35ccf45640 1 -- 172.21.15.28:0/4045721198 >> v1:172.21.15.53:6789/0 conn(0x7f35c815aaa0 legacy=0x7f35c80b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35ccf45640 1 -- 172.21.15.28:0/4045721198 >> v1:172.21.15.28:6789/0 conn(0x7f35c80d0ee0 legacy=0x7f35c80b5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.307+0000 7f35ccf45640 1 -- 172.21.15.28:0/4045721198 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c80d14a0 con 0x7f35c80affc0 2024-04-25T19:13:19.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.308+0000 7f35ccf45640 1 --2- 172.21.15.28:0/4045721198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c8167d20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009d90 tx=0x7f35c002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e24ea0c9aa8d86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:19.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.308+0000 7f35c6ffd640 1 -- 172.21.15.28:0/4045721198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c0007740 con 0x7f35c80affc0 2024-04-25T19:13:19.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.308+0000 7f35c6ffd640 1 -- 172.21.15.28:0/4045721198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35c00078e0 con 0x7f35c80affc0 2024-04-25T19:13:19.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.308+0000 7f35c6ffd640 1 -- 172.21.15.28:0/4045721198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c0007cb0 con 0x7f35c80affc0 2024-04-25T19:13:19.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.309+0000 7f35ce9a8640 1 -- 172.21.15.28:0/4045721198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 msgr2=0x7f35c8167d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.309+0000 7f35ce9a8640 1 --2- 172.21.15.28:0/4045721198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c8167d20 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009d90 tx=0x7f35c002a660 comp rx=0 tx=0).stop 2024-04-25T19:13:19.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.309+0000 7f35ce9a8640 1 -- 172.21.15.28:0/4045721198 shutdown_connections 2024-04-25T19:13:19.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.309+0000 7f35ce9a8640 1 --2- 172.21.15.28:0/4045721198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c8167d20 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.309+0000 7f35ce9a8640 1 -- 172.21.15.28:0/4045721198 >> 172.21.15.28:0/4045721198 conn(0x7f35c80742c0 msgr2=0x7f35c806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:19.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.309+0000 7f35ce9a8640 1 -- 172.21.15.28:0/4045721198 shutdown_connections 2024-04-25T19:13:19.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.309+0000 7f35ce9a8640 1 -- 172.21.15.28:0/4045721198 wait complete. 2024-04-25T19:13:19.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.311+0000 7f35ce9a8640 1 Processor -- start 2024-04-25T19:13:19.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.311+0000 7f35ce9a8640 1 -- start start 2024-04-25T19:13:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35ce9a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c813ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35ce9a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35c80d0ee0 0x7f35c80d0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35ce9a8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35c815aaa0 0x7f35c80d0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35ce9a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f35c80afaa0 con 0x7f35c80affc0 2024-04-25T19:13:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35ce9a8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f35c80af7c0 con 0x7f35c815aaa0 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35ce9a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f35c8168260 con 0x7f35c80d0ee0 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c813ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35c7fff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c813ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59660/0 (socket says 172.21.15.28:59660) 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.312+0000 7f35c7fff640 1 -- 172.21.15.28:0/79088744 learned_addr learned my addr 172.21.15.28:0/79088744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c7fff640 1 -- 172.21.15.28:0/79088744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35c80d0ee0 msgr2=0x7f35c80d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c7fff640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35c80d0ee0 0x7f35c80d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c7fff640 1 -- 172.21.15.28:0/79088744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35c815aaa0 msgr2=0x7f35c80d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c7fff640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35c815aaa0 0x7f35c80d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c7fff640 1 -- 172.21.15.28:0/79088744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35c815e7b0 con 0x7f35c80affc0 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c7fff640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c813ff60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f35b800c3f0 tx=0x7f35b800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c4ff9640 1 -- 172.21.15.28:0/79088744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b80052d0 con 0x7f35c80affc0 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c4ff9640 1 -- 172.21.15.28:0/79088744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35b8005490 con 0x7f35c80affc0 2024-04-25T19:13:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35c815ea90 con 0x7f35c80affc0 2024-04-25T19:13:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35c4ff9640 1 -- 172.21.15.28:0/79088744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b8007260 con 0x7f35c80affc0 2024-04-25T19:13:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.313+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35c815f050 con 0x7f35c80affc0 2024-04-25T19:13:19.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.314+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35c8050740 con 0x7f35c80affc0 2024-04-25T19:13:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.315+0000 7f35c4ff9640 1 -- 172.21.15.28:0/79088744 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f35b8007400 con 0x7f35c80affc0 2024-04-25T19:13:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.315+0000 7f35c4ff9640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f35a80744b0 0x7f35a80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.316+0000 7f35c77fe640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f35a80744b0 0x7f35a80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.316+0000 7f35c4ff9640 1 -- 172.21.15.28:0/79088744 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f35b8094b50 con 0x7f35c80affc0 2024-04-25T19:13:19.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.316+0000 7f35c77fe640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f35a80744b0 0x7f35a80769c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f35b0007fd0 tx=0x7f35b0001150 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:19.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.320+0000 7f35c4ff9640 1 -- 172.21.15.28:0/79088744 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35b805ee80 con 0x7f35c80affc0 2024-04-25T19:13:19.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.506+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) -- 0x7f35c8070f20 con 0x7f35c80affc0 2024-04-25T19:13:19.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.506+0000 7f35c4ff9640 1 -- 172.21.15.28:0/79088744 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v48) ==== 109+0+11 (secure 0 0 0) 0x7f35b805e780 con 0x7f35c80affc0 2024-04-25T19:13:19.508 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:13:19.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f35a80744b0 msgr2=0x7f35a80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f35a80744b0 0x7f35a80769c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f35b0007fd0 tx=0x7f35b0001150 comp rx=0 tx=0).stop 2024-04-25T19:13:19.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 msgr2=0x7f35c813ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c813ff60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f35b800c3f0 tx=0x7f35b800c8b0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 shutdown_connections 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f35a80744b0 0x7f35a80769c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f35c815aaa0 0x7f35c80d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f35c80d0ee0 0x7f35c80d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 --2- 172.21.15.28:0/79088744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35c80affc0 0x7f35c813ff60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 >> 172.21.15.28:0/79088744 conn(0x7f35c80742c0 msgr2=0x7f35c8156f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 shutdown_connections 2024-04-25T19:13:19.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.508+0000 7f35ce9a8640 1 -- 172.21.15.28:0/79088744 wait complete. 2024-04-25T19:13:19.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T19:13:19.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.633+0000 7f622f7c5640 1 Processor -- start 2024-04-25T19:13:19.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.634+0000 7f622f7c5640 1 -- start start 2024-04-25T19:13:19.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622f7c5640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6228160310 0x7f62280d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622f7c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b9920 0x7f62280b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622f7c5640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6228160720 con 0x7f62280b79f0 2024-04-25T19:13:19.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622f7c5640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f62280d5640 con 0x7f6228160310 2024-04-25T19:13:19.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622f7c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f62280d0f40 con 0x7f62280b9920 2024-04-25T19:13:19.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622d561640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6228160310 0x7f62280d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622dd62640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b9920 0x7f62280b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622d561640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6228160310 0x7f62280d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:55160/0 (socket says 172.21.15.28:55160) 2024-04-25T19:13:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622dd62640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b9920 0x7f62280b9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:43452/0 (socket says 172.21.15.28:43452) 2024-04-25T19:13:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622d561640 1 -- 172.21.15.28:0/3938814466 learned_addr learned my addr 172.21.15.28:0/3938814466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.635+0000 7f622cd60640 1 --1- 172.21.15.28:0/3938814466 >> v1:172.21.15.53:6789/0 conn(0x7f62280b79f0 0x7f62280b7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:59710/0 (socket says 172.21.15.28:59710) 2024-04-25T19:13:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.636+0000 7f622d561640 1 -- 172.21.15.28:0/3938814466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b9920 msgr2=0x7f62280b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.636+0000 7f622d561640 1 --2- 172.21.15.28:0/3938814466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b9920 0x7f62280b9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.636+0000 7f622d561640 1 -- 172.21.15.28:0/3938814466 >> v1:172.21.15.53:6789/0 conn(0x7f62280b79f0 legacy=0x7f62280b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.636+0000 7f622d561640 1 -- 172.21.15.28:0/3938814466 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62280d1220 con 0x7f6228160310 2024-04-25T19:13:19.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.636+0000 7f622dd62640 1 --2- 172.21.15.28:0/3938814466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b9920 0x7f62280b9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.636+0000 7f622d561640 1 --2- 172.21.15.28:0/3938814466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6228160310 0x7f62280d5100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6220009870 tx=0x7f622002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3715e45c2068c06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f6227fff640 1 -- 172.21.15.28:0/3938814466 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220037070 con 0x7f6228160310 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f6227fff640 1 -- 172.21.15.28:0/3938814466 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6220032440 con 0x7f6228160310 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f6227fff640 1 -- 172.21.15.28:0/3938814466 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6220032810 con 0x7f6228160310 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f622f7c5640 1 -- 172.21.15.28:0/3938814466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6228160310 msgr2=0x7f62280d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f622f7c5640 1 --2- 172.21.15.28:0/3938814466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6228160310 0x7f62280d5100 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6220009870 tx=0x7f622002a270 comp rx=0 tx=0).stop 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f622f7c5640 1 -- 172.21.15.28:0/3938814466 shutdown_connections 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f622f7c5640 1 --2- 172.21.15.28:0/3938814466 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b9920 0x7f62280b9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f622f7c5640 1 --2- 172.21.15.28:0/3938814466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6228160310 0x7f62280d5100 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.637+0000 7f622f7c5640 1 -- 172.21.15.28:0/3938814466 >> 172.21.15.28:0/3938814466 conn(0x7f62280d2a50 msgr2=0x7f62280d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.638+0000 7f622f7c5640 1 -- 172.21.15.28:0/3938814466 shutdown_connections 2024-04-25T19:13:19.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.638+0000 7f622f7c5640 1 -- 172.21.15.28:0/3938814466 wait complete. 2024-04-25T19:13:19.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.639+0000 7f622f7c5640 1 Processor -- start 2024-04-25T19:13:19.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.639+0000 7f622f7c5640 1 -- start start 2024-04-25T19:13:19.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622f7c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b79f0 0x7f62281514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622f7c5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62280b9920 0x7f62280ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622f7c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228160310 0x7f62280b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622f7c5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f62280d0bf0 con 0x7f6228160310 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622f7c5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f62280b9400 con 0x7f62280b9920 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622f7c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f62280cc950 con 0x7f62280b79f0 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228160310 0x7f62280b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228160310 0x7f62280b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59664/0 (socket says 172.21.15.28:59664) 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 -- 172.21.15.28:0/1296103388 learned_addr learned my addr 172.21.15.28:0/1296103388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622d561640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b79f0 0x7f62281514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622cd60640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62280b9920 0x7f62280ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 -- 172.21.15.28:0/1296103388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b79f0 msgr2=0x7f62281514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b79f0 0x7f62281514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 -- 172.21.15.28:0/1296103388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62280b9920 msgr2=0x7f62280ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62280b9920 0x7f62280ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 -- 172.21.15.28:0/1296103388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62280b5860 con 0x7f6228160310 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622dd62640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228160310 0x7f62280b6900 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f621800efc0 tx=0x7f621800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f6225ffb640 1 -- 172.21.15.28:0/1296103388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6218018030 con 0x7f6228160310 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f6225ffb640 1 -- 172.21.15.28:0/1296103388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f621800cd30 con 0x7f6228160310 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62280b5b40 con 0x7f6228160310 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.641+0000 7f6225ffb640 1 -- 172.21.15.28:0/1296103388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6218003bb0 con 0x7f6228160310 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.642+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62280b6100 con 0x7f6228160310 2024-04-25T19:13:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.643+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f622814e860 con 0x7f6228160310 2024-04-25T19:13:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.643+0000 7f6225ffb640 1 -- 172.21.15.28:0/1296103388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f621801b070 con 0x7f6228160310 2024-04-25T19:13:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.644+0000 7f6225ffb640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62140744b0 0x7f62140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.644+0000 7f6225ffb640 1 -- 172.21.15.28:0/1296103388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f6218095f50 con 0x7f6228160310 2024-04-25T19:13:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.644+0000 7f622d561640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62140744b0 0x7f62140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.644+0000 7f622d561640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62140744b0 0x7f62140769c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6220034000 tx=0x7f622002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:19.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.649+0000 7f6225ffb640 1 -- 172.21.15.28:0/1296103388 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f622814e860 con 0x7f6228160310 2024-04-25T19:13:19.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.845+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f62280c90d0 con 0x7f6228160310 2024-04-25T19:13:19.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.845+0000 7f6225ffb640 1 -- 172.21.15.28:0/1296103388 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v48) ==== 80+0+120 (secure 0 0 0) 0x7f6218010040 con 0x7f6228160310 2024-04-25T19:13:19.848 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.53:6835/803470605 2024-04-26T19:12:54.688887+0000 2024-04-25T19:13:19.848 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.53:6833/803470605 2024-04-26T19:12:54.688887+0000 2024-04-25T19:13:19.848 INFO:teuthology.orchestra.run.smithi028.stderr:listed 2 entries 2024-04-25T19:13:19.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62140744b0 msgr2=0x7f62140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62140744b0 0x7f62140769c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6220034000 tx=0x7f622002e000 comp rx=0 tx=0).stop 2024-04-25T19:13:19.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228160310 msgr2=0x7f62280b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:19.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228160310 0x7f62280b6900 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f621800efc0 tx=0x7f621800c490 comp rx=0 tx=0).stop 2024-04-25T19:13:19.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 shutdown_connections 2024-04-25T19:13:19.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f62140744b0 0x7f62140769c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6228160310 0x7f62280b6900 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62280b9920 0x7f62280ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 --2- 172.21.15.28:0/1296103388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f62280b79f0 0x7f62281514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 >> 172.21.15.28:0/1296103388 conn(0x7f62280d2a50 msgr2=0x7f622815f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 shutdown_connections 2024-04-25T19:13:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:19.848+0000 7f622f7c5640 1 -- 172.21.15.28:0/1296103388 wait complete. 2024-04-25T19:13:19.861 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T19:13:19.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:19.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T19:13:19.890 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:13:19.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:19.891 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T19:13:19.947 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T19:13:19.947 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:13:19.948 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:19.967 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=889 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T19:13:19.967 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T19:13:19.967 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:13:19.967 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:20.003 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16641 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T19:13:20.003 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T19:13:20.004 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:13:20.004 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:20.024 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=889 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T19:13:20.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:20.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.153+0000 7f9bdeebd640 1 Processor -- start 2024-04-25T19:13:20.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.153+0000 7f9bdeebd640 1 -- start start 2024-04-25T19:13:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bdeebd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9bd809cb60 0x7f9bd80cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bdeebd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd80cef50 0x7f9bd80d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bdeebd640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bd806d710 con 0x7f9bd809d010 2024-04-25T19:13:20.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bdeebd640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f9bd808cc20 con 0x7f9bd809cb60 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bdeebd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd808ce70 con 0x7f9bd80cef50 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bd7fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9bd809cb60 0x7f9bd80cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bd7fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9bd809cb60 0x7f9bd80cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59672/0 (socket says 172.21.15.28:59672) 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bdcc59640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9bd809d010 0x7f9bd806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37032/0 (socket says 172.21.15.28:37032) 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bd7fff640 1 -- 172.21.15.28:0/407791196 learned_addr learned my addr 172.21.15.28:0/407791196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bdd45a640 1 --2- 172.21.15.28:0/407791196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd80cef50 0x7f9bd80d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bd7fff640 1 -- 172.21.15.28:0/407791196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd80cef50 msgr2=0x7f9bd80d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bd7fff640 1 --2- 172.21.15.28:0/407791196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd80cef50 0x7f9bd80d1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bd77fe640 1 -- 172.21.15.28:0/407791196 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3951092512 0 0) 0x7f9bd806d710 con 0x7f9bd809d010 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bd7fff640 1 -- 172.21.15.28:0/407791196 >> v1:172.21.15.28:6789/0 conn(0x7f9bd809d010 legacy=0x7f9bd806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.155+0000 7f9bd7fff640 1 -- 172.21.15.28:0/407791196 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd80b92d0 con 0x7f9bd809cb60 2024-04-25T19:13:20.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.156+0000 7f9bd7fff640 1 --2- 172.21.15.28:0/407791196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80cea10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009870 tx=0x7f9bcc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12af0f50fb71353b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.156+0000 7f9bd77fe640 1 -- 172.21.15.28:0/407791196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc037070 con 0x7f9bd809cb60 2024-04-25T19:13:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.156+0000 7f9bd77fe640 1 -- 172.21.15.28:0/407791196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bcc032440 con 0x7f9bd809cb60 2024-04-25T19:13:20.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.156+0000 7f9bd77fe640 1 -- 172.21.15.28:0/407791196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bcc032810 con 0x7f9bd809cb60 2024-04-25T19:13:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.157+0000 7f9bdeebd640 1 -- 172.21.15.28:0/407791196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 msgr2=0x7f9bd80cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.157+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/407791196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80cea10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc009870 tx=0x7f9bcc02a270 comp rx=0 tx=0).stop 2024-04-25T19:13:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.157+0000 7f9bdeebd640 1 -- 172.21.15.28:0/407791196 shutdown_connections 2024-04-25T19:13:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.157+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/407791196 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd80cef50 0x7f9bd80d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.157+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/407791196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80cea10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.157+0000 7f9bdeebd640 1 -- 172.21.15.28:0/407791196 >> 172.21.15.28:0/407791196 conn(0x7f9bd80d0430 msgr2=0x7f9bd80d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:20.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.157+0000 7f9bdeebd640 1 -- 172.21.15.28:0/407791196 shutdown_connections 2024-04-25T19:13:20.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.157+0000 7f9bdeebd640 1 -- 172.21.15.28:0/407791196 wait complete. 2024-04-25T19:13:20.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.159+0000 7f9bdeebd640 1 Processor -- start 2024-04-25T19:13:20.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.159+0000 7f9bdeebd640 1 -- start start 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.160+0000 7f9bdeebd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.160+0000 7f9bdeebd640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd809d010 0x7f9bd81f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.160+0000 7f9bdeebd640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bd80cef50 0x7f9bd81f6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.160+0000 7f9bdeebd640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd806ea20 con 0x7f9bd809cb60 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.160+0000 7f9bdeebd640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9bd806e5e0 con 0x7f9bd80cef50 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.160+0000 7f9bdeebd640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9bd806eba0 con 0x7f9bd809d010 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.160+0000 7f9bdcc59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.160+0000 7f9bdcc59640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80a0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59688/0 (socket says 172.21.15.28:59688) 2024-04-25T19:13:20.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdcc59640 1 -- 172.21.15.28:0/1091450006 learned_addr learned my addr 172.21.15.28:0/1091450006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bd7fff640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd809d010 0x7f9bd81f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdd45a640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bd80cef50 0x7f9bd81f6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdcc59640 1 -- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd809d010 msgr2=0x7f9bd81f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdcc59640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd809d010 0x7f9bd81f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdcc59640 1 -- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bd80cef50 msgr2=0x7f9bd81f6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdcc59640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bd80cef50 0x7f9bd81f6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdcc59640 1 -- 172.21.15.28:0/1091450006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bd80a0b90 con 0x7f9bd809cb60 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bd7fff640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd809d010 0x7f9bd81f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdd45a640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bd80cef50 0x7f9bd81f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdcc59640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80a0f60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000c7d0 tx=0x7f9bd000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bd57fa640 1 -- 172.21.15.28:0/1091450006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd000a5e0 con 0x7f9bd809cb60 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bd57fa640 1 -- 172.21.15.28:0/1091450006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bd000a780 con 0x7f9bd809cb60 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bd80cbcc0 con 0x7f9bd809cb60 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bd57fa640 1 -- 172.21.15.28:0/1091450006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bd000ab70 con 0x7f9bd809cb60 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.161+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bd80cc280 con 0x7f9bd809cb60 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.162+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bd8050740 con 0x7f9bd809cb60 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.163+0000 7f9bd57fa640 1 -- 172.21.15.28:0/1091450006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9bd000ad10 con 0x7f9bd809cb60 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.163+0000 7f9bd57fa640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9bc40744e0 0x7f9bc40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.163+0000 7f9bd57fa640 1 -- 172.21.15.28:0/1091450006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f9bd0095520 con 0x7f9bd809cb60 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.163+0000 7f9bd7fff640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9bc40744e0 0x7f9bc40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.164+0000 7f9bd7fff640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9bc40744e0 0x7f9bc40769f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc034000 tx=0x7f9bcc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.168+0000 7f9bd57fa640 1 -- 172.21.15.28:0/1091450006 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9bd005f8f0 con 0x7f9bd809cb60 2024-04-25T19:13:20.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.392+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bd80cdd30 con 0x7f9bd809cb60 2024-04-25T19:13:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.393+0000 7f9bd57fa640 1 -- 172.21.15.28:0/1091450006 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6483 (secure 0 0 0) 0x7f9bd005f1f0 con 0x7f9bd809cb60 2024-04-25T19:13:20.395 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:20.395 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:20.395 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-04-25T19:13:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9bc40744e0 msgr2=0x7f9bc40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9bc40744e0 0x7f9bc40769f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc034000 tx=0x7f9bcc02e000 comp rx=0 tx=0).stop 2024-04-25T19:13:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 msgr2=0x7f9bd80a0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80a0f60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000c7d0 tx=0x7f9bd000cc90 comp rx=0 tx=0).stop 2024-04-25T19:13:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 shutdown_connections 2024-04-25T19:13:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9bc40744e0 0x7f9bc40769f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bd80cef50 0x7f9bd81f6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9bd809d010 0x7f9bd81f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 --2- 172.21.15.28:0/1091450006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd809cb60 0x7f9bd80a0f60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 >> 172.21.15.28:0/1091450006 conn(0x7f9bd80d0430 msgr2=0x7f9bd80b7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 shutdown_connections 2024-04-25T19:13:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.395+0000 7f9bdeebd640 1 -- 172.21.15.28:0/1091450006 wait complete. 2024-04-25T19:13:20.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.518+0000 7f8d25e68640 1 Processor -- start 2024-04-25T19:13:20.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.518+0000 7f8d25e68640 1 -- start start 2024-04-25T19:13:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d25e68640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d2009cb60 0x7f8d200cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d25e68640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d200cef50 0x7f8d200d1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d25e68640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d2006d710 con 0x7f8d2009d010 2024-04-25T19:13:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d25e68640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f8d2008cc20 con 0x7f8d2009cb60 2024-04-25T19:13:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d25e68640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8d2008ce70 con 0x7f8d200cef50 2024-04-25T19:13:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d1effd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d2009cb60 0x7f8d200cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d1effd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d2009cb60 0x7f8d200cea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59700/0 (socket says 172.21.15.28:59700) 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d1effd640 1 -- 172.21.15.28:0/644972645 learned_addr learned my addr 172.21.15.28:0/644972645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d1ffff640 1 --2- 172.21.15.28:0/644972645 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d200cef50 0x7f8d200d1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d1effd640 1 -- 172.21.15.28:0/644972645 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d200cef50 msgr2=0x7f8d200d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d1effd640 1 --2- 172.21.15.28:0/644972645 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d200cef50 0x7f8d200d1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d1effd640 1 -- 172.21.15.28:0/644972645 >> v1:172.21.15.53:6789/0 conn(0x7f8d2009d010 legacy=0x7f8d2006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.520+0000 7f8d1effd640 1 -- 172.21.15.28:0/644972645 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d200b92d0 con 0x7f8d2009cb60 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.521+0000 7f8d1effd640 1 --2- 172.21.15.28:0/644972645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009cb60 0x7f8d200cea10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8d200691b0 tx=0x7f8d1402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cba1ca46a76d0f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.521+0000 7f8d1e7fc640 1 -- 172.21.15.28:0/644972645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14037070 con 0x7f8d2009cb60 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.521+0000 7f8d1e7fc640 1 -- 172.21.15.28:0/644972645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d14032440 con 0x7f8d2009cb60 2024-04-25T19:13:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.521+0000 7f8d1e7fc640 1 -- 172.21.15.28:0/644972645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14032810 con 0x7f8d2009cb60 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.522+0000 7f8d25e68640 1 -- 172.21.15.28:0/644972645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009cb60 msgr2=0x7f8d200cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.522+0000 7f8d25e68640 1 --2- 172.21.15.28:0/644972645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009cb60 0x7f8d200cea10 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8d200691b0 tx=0x7f8d1402a1e0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.522+0000 7f8d25e68640 1 -- 172.21.15.28:0/644972645 shutdown_connections 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.522+0000 7f8d25e68640 1 --2- 172.21.15.28:0/644972645 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d200cef50 0x7f8d200d1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.522+0000 7f8d25e68640 1 --2- 172.21.15.28:0/644972645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009cb60 0x7f8d200cea10 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.522+0000 7f8d25e68640 1 -- 172.21.15.28:0/644972645 >> 172.21.15.28:0/644972645 conn(0x7f8d200d0430 msgr2=0x7f8d200d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.522+0000 7f8d25e68640 1 -- 172.21.15.28:0/644972645 shutdown_connections 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.522+0000 7f8d25e68640 1 -- 172.21.15.28:0/644972645 wait complete. 2024-04-25T19:13:20.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.524+0000 7f8d25e68640 1 Processor -- start 2024-04-25T19:13:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.524+0000 7f8d25e68640 1 -- start start 2024-04-25T19:13:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d25e68640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d2009cb60 0x7f8d201ea700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d25e68640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009d010 0x7f8d201eac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d25e68640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d200cef50 0x7f8d201eb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d25e68640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2006ea20 con 0x7f8d2009d010 2024-04-25T19:13:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d25e68640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2006e5e0 con 0x7f8d200cef50 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d25e68640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f8d2006eba0 con 0x7f8d2009cb60 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d1effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009d010 0x7f8d201eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d1effd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009d010 0x7f8d201eac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59708/0 (socket says 172.21.15.28:59708) 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.525+0000 7f8d1effd640 1 -- 172.21.15.28:0/3122830305 learned_addr learned my addr 172.21.15.28:0/3122830305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d1effd640 1 -- 172.21.15.28:0/3122830305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d2009cb60 msgr2=0x7f8d201ea700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d1f7fe640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d2009cb60 0x7f8d201ea700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d1effd640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d2009cb60 0x7f8d201ea700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d1ffff640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d200cef50 0x7f8d201eb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d1effd640 1 -- 172.21.15.28:0/3122830305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d200cef50 msgr2=0x7f8d201eb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d1effd640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d200cef50 0x7f8d201eb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d1effd640 1 -- 172.21.15.28:0/3122830305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d200cdc40 con 0x7f8d2009d010 2024-04-25T19:13:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d1effd640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009d010 0x7f8d201eac40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8d14000c00 tx=0x7f8d140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d07fff640 1 -- 172.21.15.28:0/3122830305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14041070 con 0x7f8d2009d010 2024-04-25T19:13:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d07fff640 1 -- 172.21.15.28:0/3122830305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d1403c870 con 0x7f8d2009d010 2024-04-25T19:13:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d200b21d0 con 0x7f8d2009d010 2024-04-25T19:13:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d07fff640 1 -- 172.21.15.28:0/3122830305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d14037040 con 0x7f8d2009d010 2024-04-25T19:13:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.526+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d200b2790 con 0x7f8d2009d010 2024-04-25T19:13:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.527+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d200cf360 con 0x7f8d2009d010 2024-04-25T19:13:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.528+0000 7f8d07fff640 1 -- 172.21.15.28:0/3122830305 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8d14044070 con 0x7f8d2009d010 2024-04-25T19:13:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.528+0000 7f8d07fff640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d0c0744e0 0x7f8d0c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.528+0000 7f8d07fff640 1 -- 172.21.15.28:0/3122830305 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f8d140b60b0 con 0x7f8d2009d010 2024-04-25T19:13:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.528+0000 7f8d1f7fe640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d0c0744e0 0x7f8d0c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.529+0000 7f8d1f7fe640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d0c0744e0 0x7f8d0c0769f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000a790 tx=0x7f8d10008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.533+0000 7f8d07fff640 1 -- 172.21.15.28:0/3122830305 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d200cf360 con 0x7f8d2009d010 2024-04-25T19:13:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.718+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d20050740 con 0x7f8d2009d010 2024-04-25T19:13:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.720+0000 7f8d07fff640 1 -- 172.21.15.28:0/3122830305 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+18884 (secure 0 0 0) 0x7f8d14080490 con 0x7f8d2009d010 2024-04-25T19:13:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:20.722 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:10.963059+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d0c0744e0 msgr2=0x7f8d0c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d0c0744e0 0x7f8d0c0769f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000a790 tx=0x7f8d10008040 comp rx=0 tx=0).stop 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009d010 msgr2=0x7f8d201eac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009d010 0x7f8d201eac40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f8d14000c00 tx=0x7f8d140046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 shutdown_connections 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f8d0c0744e0 0x7f8d0c0769f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d200cef50 0x7f8d201eb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d2009d010 0x7f8d201eac40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 --2- 172.21.15.28:0/3122830305 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f8d2009cb60 0x7f8d201ea700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 >> 172.21.15.28:0/3122830305 conn(0x7f8d200d0430 msgr2=0x7f8d200b6800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 shutdown_connections 2024-04-25T19:13:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.722+0000 7f8d25e68640 1 -- 172.21.15.28:0/3122830305 wait complete. 2024-04-25T19:13:20.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T19:12:41.809696+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:12:43.014197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-04-25T19:13:02.679639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:13:20.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:13:20.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.849+0000 7fede60fa640 1 Processor -- start 2024-04-25T19:13:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.849+0000 7fede60fa640 1 -- start start 2024-04-25T19:13:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7fede60fa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede006ed00 0x7fede00d3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7fede60fa640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fede00ccf20 0x7fede00cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7fede60fa640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fede0161990 con 0x7fede00cd870 2024-04-25T19:13:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7fede60fa640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fede00d2dc0 con 0x7fede00ccf20 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7fede60fa640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fede00d24b0 con 0x7fede006ed00 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7feddeffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fede00ccf20 0x7fede00cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7feddffff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fede00cd870 0x7fede00b3ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39780/0 (socket says 172.21.15.28:39780) 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7feddeffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fede00ccf20 0x7fede00cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35722/0 (socket says 172.21.15.28:35722) 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7feddffff640 1 -- 172.21.15.28:0/3315361551 learned_addr learned my addr 172.21.15.28:0/3315361551 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7feddf7fe640 1 --2- 172.21.15.28:0/3315361551 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede006ed00 0x7fede00d3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7feddeffd640 1 -- 172.21.15.28:0/3315361551 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede006ed00 msgr2=0x7fede00d3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7feddeffd640 1 --2- 172.21.15.28:0/3315361551 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede006ed00 0x7fede00d3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7feddeffd640 1 -- 172.21.15.28:0/3315361551 >> v1:172.21.15.28:6789/0 conn(0x7fede00cd870 legacy=0x7fede00b3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.851+0000 7feddeffd640 1 -- 172.21.15.28:0/3315361551 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede00d2770 con 0x7fede00ccf20 2024-04-25T19:13:20.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.850+0000 7fedde7fc640 1 -- 172.21.15.28:0/3315361551 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2285005427 0 0) 0x7fede0161990 con 0x7fede00cd870 2024-04-25T19:13:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.851+0000 7feddeffd640 1 --2- 172.21.15.28:0/3315361551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede00ccf20 0x7fede00cd330 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fedd40097a0 tx=0x7fedd402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2449b4d1b4d62194 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.851+0000 7fedde7fc640 1 -- 172.21.15.28:0/3315361551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4037070 con 0x7fede00ccf20 2024-04-25T19:13:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.851+0000 7fedde7fc640 1 -- 172.21.15.28:0/3315361551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedd4032440 con 0x7fede00ccf20 2024-04-25T19:13:20.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.851+0000 7fedde7fc640 1 -- 172.21.15.28:0/3315361551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4032810 con 0x7fede00ccf20 2024-04-25T19:13:20.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.852+0000 7fede60fa640 1 -- 172.21.15.28:0/3315361551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede00ccf20 msgr2=0x7fede00cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.852+0000 7fede60fa640 1 --2- 172.21.15.28:0/3315361551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede00ccf20 0x7fede00cd330 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fedd40097a0 tx=0x7fedd402a1a0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.852+0000 7fede60fa640 1 -- 172.21.15.28:0/3315361551 shutdown_connections 2024-04-25T19:13:20.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.852+0000 7fede60fa640 1 --2- 172.21.15.28:0/3315361551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede00ccf20 0x7fede00cd330 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.852+0000 7fede60fa640 1 --2- 172.21.15.28:0/3315361551 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede006ed00 0x7fede00d3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.852+0000 7fede60fa640 1 -- 172.21.15.28:0/3315361551 >> 172.21.15.28:0/3315361551 conn(0x7fede00b7640 msgr2=0x7fede00b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.852+0000 7fede60fa640 1 -- 172.21.15.28:0/3315361551 shutdown_connections 2024-04-25T19:13:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.852+0000 7fede60fa640 1 -- 172.21.15.28:0/3315361551 wait complete. 2024-04-25T19:13:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.854+0000 7fede60fa640 1 Processor -- start 2024-04-25T19:13:20.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.854+0000 7fede60fa640 1 -- start start 2024-04-25T19:13:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.855+0000 7fede60fa640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede006ed00 0x7fede0159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.855+0000 7fede60fa640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede00ccf20 0x7fede01f99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fede60fa640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fede00cd870 0x7fede0157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fede60fa640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fede00cf290 con 0x7fede006ed00 2024-04-25T19:13:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fede60fa640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fede00cf110 con 0x7fede00cd870 2024-04-25T19:13:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fede60fa640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fede01614d0 con 0x7fede00ccf20 2024-04-25T19:13:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede006ed00 0x7fede0159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede006ed00 0x7fede0159590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35730/0 (socket says 172.21.15.28:35730) 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 -- 172.21.15.28:0/3060528079 learned_addr learned my addr 172.21.15.28:0/3060528079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddeffd640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede00ccf20 0x7fede01f99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddffff640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fede00cd870 0x7fede0157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 -- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede00ccf20 msgr2=0x7fede01f99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede00ccf20 0x7fede01f99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 -- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fede00cd870 msgr2=0x7fede0157a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fede00cd870 0x7fede0157a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 -- 172.21.15.28:0/3060528079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede01fa460 con 0x7fede006ed00 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddffff640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fede00cd870 0x7fede0157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7feddf7fe640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede006ed00 0x7fede0159590 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fedd000d5f0 tx=0x7fedd000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fedc7fff640 1 -- 172.21.15.28:0/3060528079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd00125f0 con 0x7fede006ed00 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fedc7fff640 1 -- 172.21.15.28:0/3060528079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fedd0019070 con 0x7fede006ed00 2024-04-25T19:13:20.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede01fa740 con 0x7fede006ed00 2024-04-25T19:13:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fedc7fff640 1 -- 172.21.15.28:0/3060528079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd00129d0 con 0x7fede006ed00 2024-04-25T19:13:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.856+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede0158d90 con 0x7fede006ed00 2024-04-25T19:13:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.857+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede00d0210 con 0x7fede006ed00 2024-04-25T19:13:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.858+0000 7fedc7fff640 1 -- 172.21.15.28:0/3060528079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fedd0012b70 con 0x7fede006ed00 2024-04-25T19:13:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.858+0000 7fedc7fff640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fedcc0744e0 0x7fedcc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.859+0000 7fedc7fff640 1 -- 172.21.15.28:0/3060528079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fedd0095a10 con 0x7fede006ed00 2024-04-25T19:13:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.859+0000 7feddeffd640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fedcc0744e0 0x7fedcc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.859+0000 7feddeffd640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fedcc0744e0 0x7fedcc0769f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fedd4034000 tx=0x7fedd402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:20.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:20.863+0000 7fedc7fff640 1 -- 172.21.15.28:0/3060528079 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fede00d0210 con 0x7fede006ed00 2024-04-25T19:13:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.048+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fede0050740 con 0x7fede006ed00 2024-04-25T19:13:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.048+0000 7fedc7fff640 1 -- 172.21.15.28:0/3060528079 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7fedd005fde0 con 0x7fede006ed00 2024-04-25T19:13:21.050 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.050+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fedcc0744e0 msgr2=0x7fedcc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.050+0000 7fede60fa640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fedcc0744e0 0x7fedcc0769f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fedd4034000 tx=0x7fedd402e000 comp rx=0 tx=0).stop 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.050+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede006ed00 msgr2=0x7fede0159590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.050+0000 7fede60fa640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede006ed00 0x7fede0159590 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fedd000d5f0 tx=0x7fedd000dab0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.051+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 shutdown_connections 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.051+0000 7fede60fa640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fedcc0744e0 0x7fedcc0769f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.051+0000 7fede60fa640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fede00cd870 0x7fede0157a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.051+0000 7fede60fa640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fede00ccf20 0x7fede01f99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.051+0000 7fede60fa640 1 --2- 172.21.15.28:0/3060528079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fede006ed00 0x7fede0159590 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.051+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 >> 172.21.15.28:0/3060528079 conn(0x7fede00b7640 msgr2=0x7fede009b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.051+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 shutdown_connections 2024-04-25T19:13:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.051+0000 7fede60fa640 1 -- 172.21.15.28:0/3060528079 wait complete. 2024-04-25T19:13:21.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T19:13:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.177+0000 7f77d2051640 1 Processor -- start 2024-04-25T19:13:21.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.177+0000 7f77d2051640 1 -- start start 2024-04-25T19:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77d2051640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77cc0b1a80 0x7f77cc0b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77d2051640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f77cc075c80 0x7f77cc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77d2051640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77cc0b1350 con 0x7f77cc06d630 2024-04-25T19:13:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77d2051640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f77cc0b15f0 con 0x7f77cc0b1a80 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77d2051640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f77cc06c610 con 0x7f77cc075c80 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77cb7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77cc0b1a80 0x7f77cc0b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77cb7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f77cc0b1a80 0x7f77cc0b0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35742/0 (socket says 172.21.15.28:35742) 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77cb7fe640 1 -- 172.21.15.28:0/68987629 learned_addr learned my addr 172.21.15.28:0/68987629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77caffd640 1 --2- 172.21.15.28:0/68987629 >> v2:172.21.15.53:3300/0 conn(0x7f77cc075c80 0x7f77cc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77cb7fe640 1 -- 172.21.15.28:0/68987629 >> v2:172.21.15.53:3300/0 conn(0x7f77cc075c80 msgr2=0x7f77cc06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77cb7fe640 1 --2- 172.21.15.28:0/68987629 >> v2:172.21.15.53:3300/0 conn(0x7f77cc075c80 0x7f77cc06d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77cb7fe640 1 -- 172.21.15.28:0/68987629 >> v1:172.21.15.53:6789/0 conn(0x7f77cc06d630 legacy=0x7f77cc15d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.179+0000 7f77cb7fe640 1 -- 172.21.15.28:0/68987629 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc06c8f0 con 0x7f77cc0b1a80 2024-04-25T19:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.180+0000 7f77cb7fe640 1 --2- 172.21.15.28:0/68987629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc0b0d20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f77cc0742a0 tx=0x7f77bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f09ed4fed4ebd7aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.180+0000 7f77ca7fc640 1 -- 172.21.15.28:0/68987629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc037070 con 0x7f77cc0b1a80 2024-04-25T19:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.180+0000 7f77ca7fc640 1 -- 172.21.15.28:0/68987629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77bc032440 con 0x7f77cc0b1a80 2024-04-25T19:13:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.180+0000 7f77ca7fc640 1 -- 172.21.15.28:0/68987629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77bc032810 con 0x7f77cc0b1a80 2024-04-25T19:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.181+0000 7f77d2051640 1 -- 172.21.15.28:0/68987629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 msgr2=0x7f77cc0b0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.181+0000 7f77d2051640 1 --2- 172.21.15.28:0/68987629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc0b0d20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f77cc0742a0 tx=0x7f77bc02a270 comp rx=0 tx=0).stop 2024-04-25T19:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.181+0000 7f77d2051640 1 -- 172.21.15.28:0/68987629 shutdown_connections 2024-04-25T19:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.181+0000 7f77d2051640 1 --2- 172.21.15.28:0/68987629 >> v2:172.21.15.53:3300/0 conn(0x7f77cc075c80 0x7f77cc06d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.181+0000 7f77d2051640 1 --2- 172.21.15.28:0/68987629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc0b0d20 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.181+0000 7f77d2051640 1 -- 172.21.15.28:0/68987629 >> 172.21.15.28:0/68987629 conn(0x7f77cc073d30 msgr2=0x7f77cc09b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.181+0000 7f77d2051640 1 -- 172.21.15.28:0/68987629 shutdown_connections 2024-04-25T19:13:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.181+0000 7f77d2051640 1 -- 172.21.15.28:0/68987629 wait complete. 2024-04-25T19:13:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.183+0000 7f77d2051640 1 Processor -- start 2024-04-25T19:13:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.183+0000 7f77d2051640 1 -- start start 2024-04-25T19:13:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77d2051640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77cc06d630 0x7f77cc1465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77d2051640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77cc075c80 0x7f77cc147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77d2051640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc1f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77d2051640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc076090 con 0x7f77cc0b1a80 2024-04-25T19:13:21.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77d2051640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f77cc06da40 con 0x7f77cc06d630 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77d2051640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f77cc09db50 con 0x7f77cc075c80 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc1f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77cbfff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc1f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35744/0 (socket says 172.21.15.28:35744) 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.184+0000 7f77cbfff640 1 -- 172.21.15.28:0/224163983 learned_addr learned my addr 172.21.15.28:0/224163983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77cb7fe640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77cc06d630 0x7f77cc1465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77caffd640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77cc075c80 0x7f77cc147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77cbfff640 1 -- 172.21.15.28:0/224163983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77cc075c80 msgr2=0x7f77cc147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77cbfff640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77cc075c80 0x7f77cc147f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77cbfff640 1 -- 172.21.15.28:0/224163983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77cc06d630 msgr2=0x7f77cc1465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77cbfff640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77cc06d630 0x7f77cc1465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77cbfff640 1 -- 172.21.15.28:0/224163983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77cc1f3f50 con 0x7f77cc0b1a80 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77cbfff640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc1f3a10 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f77c400c760 tx=0x7f77c400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77b3fff640 1 -- 172.21.15.28:0/224163983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c4003b90 con 0x7f77cc0b1a80 2024-04-25T19:13:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77b3fff640 1 -- 172.21.15.28:0/224163983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77c4003d30 con 0x7f77cc0b1a80 2024-04-25T19:13:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77cc146ed0 con 0x7f77cc0b1a80 2024-04-25T19:13:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77b3fff640 1 -- 172.21.15.28:0/224163983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c400aa90 con 0x7f77cc0b1a80 2024-04-25T19:13:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.185+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77cc147490 con 0x7f77cc0b1a80 2024-04-25T19:13:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.186+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77cc146c40 con 0x7f77cc0b1a80 2024-04-25T19:13:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.187+0000 7f77b3fff640 1 -- 172.21.15.28:0/224163983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f77c400ac30 con 0x7f77cc0b1a80 2024-04-25T19:13:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.187+0000 7f77b3fff640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f77b80744b0 0x7f77b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.187+0000 7f77b3fff640 1 -- 172.21.15.28:0/224163983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f77c40952c0 con 0x7f77cc0b1a80 2024-04-25T19:13:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.187+0000 7f77cb7fe640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f77b80744b0 0x7f77b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.188+0000 7f77cb7fe640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f77b80744b0 0x7f77b80769c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f77bc034000 tx=0x7f77bc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.192+0000 7f77b3fff640 1 -- 172.21.15.28:0/224163983 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77c405f040 con 0x7f77cc0b1a80 2024-04-25T19:13:21.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.384+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f77cc1462f0 con 0x7f77cc0b1a80 2024-04-25T19:13:21.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.385+0000 7f77b3fff640 1 -- 172.21.15.28:0/224163983 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f77cc1462f0 con 0x7f77cc0b1a80 2024-04-25T19:13:21.387 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f77b80744b0 msgr2=0x7f77b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f77b80744b0 0x7f77b80769c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f77bc034000 tx=0x7f77bc02e000 comp rx=0 tx=0).stop 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 msgr2=0x7f77cc1f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc1f3a10 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f77c400c760 tx=0x7f77c400cc20 comp rx=0 tx=0).stop 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 shutdown_connections 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f77b80744b0 0x7f77b80769c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77cc0b1a80 0x7f77cc1f3a10 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f77cc075c80 0x7f77cc147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 --2- 172.21.15.28:0/224163983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77cc06d630 0x7f77cc1465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 >> 172.21.15.28:0/224163983 conn(0x7f77cc073d30 msgr2=0x7f77cc156320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:21.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 shutdown_connections 2024-04-25T19:13:21.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.387+0000 7f77d2051640 1 -- 172.21.15.28:0/224163983 wait complete. 2024-04-25T19:13:21.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:13:21.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.513+0000 7fca628b6640 1 Processor -- start 2024-04-25T19:13:21.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.513+0000 7fca628b6640 1 -- start start 2024-04-25T19:13:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.514+0000 7fca628b6640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c160250 0x7fca5c0b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.514+0000 7fca628b6640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca5c0d2ca0 0x7fca5c0d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.514+0000 7fca628b6640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca5c164b20 con 0x7fca5c0b81a0 2024-04-25T19:13:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.514+0000 7fca628b6640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fca5c0b4f50 con 0x7fca5c0d2ca0 2024-04-25T19:13:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.514+0000 7fca628b6640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fca5c163d60 con 0x7fca5c160250 2024-04-25T19:13:21.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca5b7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fca5c0b81a0 0x7fca5c0d2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39794/0 (socket says 172.21.15.28:39794) 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca5b7fe640 1 -- 172.21.15.28:0/3673798172 learned_addr learned my addr 172.21.15.28:0/3673798172 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca60e53640 1 --2- 172.21.15.28:0/3673798172 >> v2:172.21.15.53:3300/0 conn(0x7fca5c0d2ca0 0x7fca5c0d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca5bfff640 1 --2- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c160250 0x7fca5c0b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca5affd640 1 -- 172.21.15.28:0/3673798172 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 528960154 0 0) 0x7fca5c164b20 con 0x7fca5c0b81a0 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca5affd640 1 -- 172.21.15.28:0/3673798172 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca48003520 con 0x7fca5c0b81a0 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca60e53640 1 -- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c160250 msgr2=0x7fca5c0b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca60e53640 1 --2- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c160250 0x7fca5c0b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca60e53640 1 -- 172.21.15.28:0/3673798172 >> v1:172.21.15.28:6789/0 conn(0x7fca5c0b81a0 legacy=0x7fca5c0d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca5affd640 1 -- 172.21.15.28:0/3673798172 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3864955872 0 0) 0x7fca48003520 con 0x7fca5c0b81a0 2024-04-25T19:13:21.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca60e53640 1 -- 172.21.15.28:0/3673798172 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5c164040 con 0x7fca5c0d2ca0 2024-04-25T19:13:21.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.515+0000 7fca5bfff640 1 --2- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c160250 0x7fca5c0b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:21.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.516+0000 7fca60e53640 1 --2- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0d2ca0 0x7fca5c0d44b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fca54009d90 tx=0x7fca5402a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc2bf59423636cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.516+0000 7fca5affd640 1 -- 172.21.15.28:0/3673798172 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca54007740 con 0x7fca5c0d2ca0 2024-04-25T19:13:21.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.516+0000 7fca5affd640 1 -- 172.21.15.28:0/3673798172 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca540078e0 con 0x7fca5c0d2ca0 2024-04-25T19:13:21.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.516+0000 7fca5affd640 1 -- 172.21.15.28:0/3673798172 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca54007cb0 con 0x7fca5c0d2ca0 2024-04-25T19:13:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.517+0000 7fca628b6640 1 -- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0d2ca0 msgr2=0x7fca5c0d44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.517+0000 7fca628b6640 1 --2- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0d2ca0 0x7fca5c0d44b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fca54009d90 tx=0x7fca5402a5f0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.517+0000 7fca628b6640 1 -- 172.21.15.28:0/3673798172 shutdown_connections 2024-04-25T19:13:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.517+0000 7fca628b6640 1 --2- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0d2ca0 0x7fca5c0d44b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.517+0000 7fca628b6640 1 --2- 172.21.15.28:0/3673798172 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c160250 0x7fca5c0b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.517+0000 7fca628b6640 1 -- 172.21.15.28:0/3673798172 >> 172.21.15.28:0/3673798172 conn(0x7fca5c06ee30 msgr2=0x7fca5c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.517+0000 7fca628b6640 1 -- 172.21.15.28:0/3673798172 shutdown_connections 2024-04-25T19:13:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.517+0000 7fca628b6640 1 -- 172.21.15.28:0/3673798172 wait complete. 2024-04-25T19:13:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.519+0000 7fca628b6640 1 Processor -- start 2024-04-25T19:13:21.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.519+0000 7fca628b6640 1 -- start start 2024-04-25T19:13:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca628b6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0b81a0 0x7fca5c0cfe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca628b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca5c160250 0x7fca5c156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca628b6640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c1f6450 0x7fca5c157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca628b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca5c0b4bd0 con 0x7fca5c160250 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca628b6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fca5c164600 con 0x7fca5c0b81a0 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca628b6640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fca5c0d49f0 con 0x7fca5c1f6450 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca5b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca5c160250 0x7fca5c156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca5b7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca5c160250 0x7fca5c156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35750/0 (socket says 172.21.15.28:35750) 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.520+0000 7fca5b7fe640 1 -- 172.21.15.28:0/309982496 learned_addr learned my addr 172.21.15.28:0/309982496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca5b7fe640 1 -- 172.21.15.28:0/309982496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c1f6450 msgr2=0x7fca5c157f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca5b7fe640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c1f6450 0x7fca5c157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca5b7fe640 1 -- 172.21.15.28:0/309982496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0b81a0 msgr2=0x7fca5c0cfe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca5bfff640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0b81a0 0x7fca5c0cfe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca5b7fe640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0b81a0 0x7fca5c0cfe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca5b7fe640 1 -- 172.21.15.28:0/309982496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca5c156e60 con 0x7fca5c160250 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca5b7fe640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca5c160250 0x7fca5c156020 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00c410 tx=0x7fca4c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca58ff9640 1 -- 172.21.15.28:0/309982496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c017070 con 0x7fca5c160250 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca58ff9640 1 -- 172.21.15.28:0/309982496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca4c005ce0 con 0x7fca5c160250 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca5c157140 con 0x7fca5c160250 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca58ff9640 1 -- 172.21.15.28:0/309982496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c007310 con 0x7fca5c160250 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.521+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca5c157700 con 0x7fca5c160250 2024-04-25T19:13:21.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.522+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca5c156bd0 con 0x7fca5c160250 2024-04-25T19:13:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.523+0000 7fca58ff9640 1 -- 172.21.15.28:0/309982496 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fca4c01a070 con 0x7fca5c160250 2024-04-25T19:13:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.523+0000 7fca58ff9640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca3c0744b0 0x7fca3c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.523+0000 7fca5bfff640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca3c0744b0 0x7fca3c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.523+0000 7fca58ff9640 1 -- 172.21.15.28:0/309982496 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fca4c0949d0 con 0x7fca5c160250 2024-04-25T19:13:21.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.524+0000 7fca5bfff640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca3c0744b0 0x7fca3c0769c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0d05f0 tx=0x7fca44008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.528+0000 7fca58ff9640 1 -- 172.21.15.28:0/309982496 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca4c05ed90 con 0x7fca5c160250 2024-04-25T19:13:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.711+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fca5c0d0360 con 0x7fca5c160250 2024-04-25T19:13:21.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.711+0000 7fca58ff9640 1 -- 172.21.15.28:0/309982496 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7fca4c05e690 con 0x7fca5c160250 2024-04-25T19:13:21.713 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca3c0744b0 msgr2=0x7fca3c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca3c0744b0 0x7fca3c0769c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fca5c0d05f0 tx=0x7fca44008040 comp rx=0 tx=0).stop 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca5c160250 msgr2=0x7fca5c156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca5c160250 0x7fca5c156020 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fca4c00c410 tx=0x7fca4c001e70 comp rx=0 tx=0).stop 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 shutdown_connections 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fca3c0744b0 0x7fca3c0769c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fca5c1f6450 0x7fca5c157f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca5c160250 0x7fca5c156020 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 --2- 172.21.15.28:0/309982496 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5c0b81a0 0x7fca5c0cfe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 >> 172.21.15.28:0/309982496 conn(0x7fca5c06ee30 msgr2=0x7fca5c0b5570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 shutdown_connections 2024-04-25T19:13:21.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.713+0000 7fca628b6640 1 -- 172.21.15.28:0/309982496 wait complete. 2024-04-25T19:13:21.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-04-25T19:13:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.839+0000 7fc86ba30640 1 Processor -- start 2024-04-25T19:13:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.840+0000 7fc86ba30640 1 -- start start 2024-04-25T19:13:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc86ba30640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8640b21c0 0x7fc86415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc86ba30640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc864161230 0x7fc86406ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc86ba30640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8640b29a0 con 0x7fc8640d1f80 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc86ba30640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fc86406c570 con 0x7fc864161230 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc86ba30640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc8640d2540 con 0x7fc8640b21c0 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc8697cc640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc8640d1f80 0x7fc8640b1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39798/0 (socket says 172.21.15.28:39798) 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc8697cc640 1 -- 172.21.15.28:0/3420555571 learned_addr learned my addr 172.21.15.28:0/3420555571 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 31901608 0 0) 0x7fc8640b29a0 con 0x7fc8640d1f80 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc869fcd640 1 --2- 172.21.15.28:0/3420555571 >> v2:172.21.15.53:3300/0 conn(0x7fc864161230 0x7fc86406ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc84c003520 con 0x7fc8640d1f80 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.841+0000 7fc868fcb640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8640b21c0 0x7fc86415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2657673326 0 0) 0x7fc84c003520 con 0x7fc8640d1f80 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8640b21c0 msgr2=0x7fc86415dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc8640b21c0 0x7fc86415dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 >> v2:172.21.15.53:3300/0 conn(0x7fc864161230 msgr2=0x7fc86406ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 --2- 172.21.15.28:0/3420555571 >> v2:172.21.15.53:3300/0 conn(0x7fc864161230 0x7fc86406ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8640d2820 con 0x7fc8640d1f80 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fc854001a10 con 0x7fc8640d1f80 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 >> v1:172.21.15.28:6789/0 conn(0x7fc8640d1f80 legacy=0x7fc8640b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc84c0039b0 0x7fc84c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c004800 0x7fc84c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84c005150 0x7fc84c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc86406c570 con 0x7fc84c0039b0 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc8640d2540 con 0x7fc84c005150 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc84c00bf00 con 0x7fc84c004800 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc8697cc640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc84c0039b0 0x7fc84c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc868fcb640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c004800 0x7fc84c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc869fcd640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84c005150 0x7fc84c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc8697cc640 1 -- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c004800 msgr2=0x7fc84c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc8697cc640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c004800 0x7fc84c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc8697cc640 1 -- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84c005150 msgr2=0x7fc84c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc8697cc640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84c005150 0x7fc84c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc8697cc640 1 -- 172.21.15.28:0/3420555571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c003520 con 0x7fc84c0039b0 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.842+0000 7fc869fcd640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc84c005150 0x7fc84c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.843+0000 7fc8697cc640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc84c0039b0 0x7fc84c003e20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc8640b81c0 tx=0x7fc84c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4454ec5135b60a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.843+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85400a410 con 0x7fc84c0039b0 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.843+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85402f070 con 0x7fc84c0039b0 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.843+0000 7fc863fff640 1 -- 172.21.15.28:0/3420555571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85402bb90 con 0x7fc84c0039b0 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.844+0000 7fc86ba30640 1 -- 172.21.15.28:0/3420555571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc84c0039b0 msgr2=0x7fc84c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.844+0000 7fc86ba30640 1 --2- 172.21.15.28:0/3420555571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc84c0039b0 0x7fc84c003e20 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc8640b81c0 tx=0x7fc84c005830 comp rx=0 tx=0).stop 2024-04-25T19:13:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.844+0000 7fc869fcd640 1 -- 172.21.15.28:0/3420555571 reap_dead start 2024-04-25T19:13:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.844+0000 7fc86ba30640 1 -- 172.21.15.28:0/3420555571 shutdown_connections 2024-04-25T19:13:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.844+0000 7fc86ba30640 1 -- 172.21.15.28:0/3420555571 >> 172.21.15.28:0/3420555571 conn(0x7fc864073d30 msgr2=0x7fc86409b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.844+0000 7fc86ba30640 1 -- 172.21.15.28:0/3420555571 shutdown_connections 2024-04-25T19:13:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.844+0000 7fc86ba30640 1 -- 172.21.15.28:0/3420555571 wait complete. 2024-04-25T19:13:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.846+0000 7fc86ba30640 1 Processor -- start 2024-04-25T19:13:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.846+0000 7fc86ba30640 1 -- start start 2024-04-25T19:13:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc86ba30640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c0039b0 0x7fc864159560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc86ba30640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc864164000 0x7fc86415f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc86ba30640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc864164410 0x7fc86415f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc86ba30640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc864161640 con 0x7fc864164410 2024-04-25T19:13:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc86ba30640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fc8640d2de0 con 0x7fc864164000 2024-04-25T19:13:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc86ba30640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fc8640b21c0 con 0x7fc84c0039b0 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc869fcd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc864164410 0x7fc86415f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc869fcd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc864164410 0x7fc86415f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35762/0 (socket says 172.21.15.28:35762) 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc869fcd640 1 -- 172.21.15.28:0/47746290 learned_addr learned my addr 172.21.15.28:0/47746290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc8697cc640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c0039b0 0x7fc864159560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.847+0000 7fc868fcb640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc864164000 0x7fc86415f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc869fcd640 1 -- 172.21.15.28:0/47746290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c0039b0 msgr2=0x7fc864159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc869fcd640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c0039b0 0x7fc864159560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc869fcd640 1 -- 172.21.15.28:0/47746290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc864164000 msgr2=0x7fc86415f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc869fcd640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc864164000 0x7fc86415f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc869fcd640 1 -- 172.21.15.28:0/47746290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8640ceda0 con 0x7fc864164410 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc869fcd640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc864164410 0x7fc86415f8d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc86409bcd0 tx=0x7fc85c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc861ffb640 1 -- 172.21.15.28:0/47746290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c004440 con 0x7fc864164410 2024-04-25T19:13:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc861ffb640 1 -- 172.21.15.28:0/47746290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c01b070 con 0x7fc864164410 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8640afb60 con 0x7fc864164410 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc861ffb640 1 -- 172.21.15.28:0/47746290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c003850 con 0x7fc864164410 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.848+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8640b0120 con 0x7fc864164410 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.849+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8640746b0 con 0x7fc864164410 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.850+0000 7fc861ffb640 1 -- 172.21.15.28:0/47746290 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc85c0039f0 con 0x7fc864164410 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.850+0000 7fc861ffb640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc8500744b0 0x7fc8500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.851+0000 7fc861ffb640 1 -- 172.21.15.28:0/47746290 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fc85c0956f0 con 0x7fc864164410 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.851+0000 7fc8697cc640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc8500744b0 0x7fc8500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:21.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.851+0000 7fc8697cc640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc8500744b0 0x7fc8500769c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc854033000 tx=0x7fc854007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:21.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:21.855+0000 7fc861ffb640 1 -- 172.21.15.28:0/47746290 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8640746b0 con 0x7fc864164410 2024-04-25T19:13:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.039+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) -- 0x7fc8640ceb00 con 0x7fc864164410 2024-04-25T19:13:22.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.040+0000 7fc861ffb640 1 -- 172.21.15.28:0/47746290 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v48) ==== 109+0+11 (secure 0 0 0) 0x7fc85c025020 con 0x7fc864164410 2024-04-25T19:13:22.042 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc8500744b0 msgr2=0x7fc8500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc8500744b0 0x7fc8500769c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc854033000 tx=0x7fc854007000 comp rx=0 tx=0).stop 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc864164410 msgr2=0x7fc86415f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc864164410 0x7fc86415f8d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fc86409bcd0 tx=0x7fc85c009f90 comp rx=0 tx=0).stop 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 shutdown_connections 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fc8500744b0 0x7fc8500769c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc864164410 0x7fc86415f8d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc864164000 0x7fc86415f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 --2- 172.21.15.28:0/47746290 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fc84c0039b0 0x7fc864159560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 >> 172.21.15.28:0/47746290 conn(0x7fc864073d30 msgr2=0x7fc86414f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:22.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 shutdown_connections 2024-04-25T19:13:22.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.042+0000 7fc86ba30640 1 -- 172.21.15.28:0/47746290 wait complete. 2024-04-25T19:13:22.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.165+0000 7f7cc8d84640 1 Processor -- start 2024-04-25T19:13:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.165+0000 7f7cc8d84640 1 -- start start 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.166+0000 7f7cc8d84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 0x7f7cc40bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.166+0000 7f7cc8d84640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cc40b9470 0x7f7cc40b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.166+0000 7f7cc8d84640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc40bbb90 con 0x7f7cc40b7650 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.166+0000 7f7cc8d84640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f7cc40d0b50 con 0x7f7cc40b9470 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.166+0000 7f7cc8d84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc408e3a0 con 0x7f7cc40b4e30 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc1d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cc40b9470 0x7f7cc40b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc1d9b640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cc40b9470 0x7f7cc40b9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35770/0 (socket says 172.21.15.28:35770) 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc2d9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7cc40b7650 0x7f7cc40b7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39804/0 (socket says 172.21.15.28:39804) 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc1d9b640 1 -- 172.21.15.28:0/937320740 learned_addr learned my addr 172.21.15.28:0/937320740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc259c640 1 --2- 172.21.15.28:0/937320740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 0x7f7cc40bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc1d9b640 1 -- 172.21.15.28:0/937320740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 msgr2=0x7f7cc40bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc1d9b640 1 --2- 172.21.15.28:0/937320740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 0x7f7cc40bb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc1d9b640 1 -- 172.21.15.28:0/937320740 >> v1:172.21.15.28:6789/0 conn(0x7f7cc40b7650 legacy=0x7f7cc40b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc159a640 1 -- 172.21.15.28:0/937320740 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592033980 0 0) 0x7f7cbc001110 con 0x7f7cc40b7650 2024-04-25T19:13:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.167+0000 7f7cc1d9b640 1 -- 172.21.15.28:0/937320740 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc4164d60 con 0x7f7cc40b9470 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.168+0000 7f7cc1d9b640 1 --2- 172.21.15.28:0/937320740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b9470 0x7f7cc40b9880 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009870 tx=0x7f7cac02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=944c0dd94914b108 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.168+0000 7f7cc159a640 1 -- 172.21.15.28:0/937320740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac037070 con 0x7f7cc40b9470 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.168+0000 7f7cc159a640 1 -- 172.21.15.28:0/937320740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cac032440 con 0x7f7cc40b9470 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.168+0000 7f7cc159a640 1 -- 172.21.15.28:0/937320740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac032810 con 0x7f7cc40b9470 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.168+0000 7f7cc8d84640 1 -- 172.21.15.28:0/937320740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b9470 msgr2=0x7f7cc40b9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.168+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/937320740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b9470 0x7f7cc40b9880 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009870 tx=0x7f7cac02a1c0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.169+0000 7f7cc8d84640 1 -- 172.21.15.28:0/937320740 shutdown_connections 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.169+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/937320740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b9470 0x7f7cc40b9880 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.169+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/937320740 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 0x7f7cc40bb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.169+0000 7f7cc8d84640 1 -- 172.21.15.28:0/937320740 >> 172.21.15.28:0/937320740 conn(0x7f7cc40d2650 msgr2=0x7f7cc40d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.169+0000 7f7cc8d84640 1 -- 172.21.15.28:0/937320740 shutdown_connections 2024-04-25T19:13:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.169+0000 7f7cc8d84640 1 -- 172.21.15.28:0/937320740 wait complete. 2024-04-25T19:13:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.170+0000 7f7cc8d84640 1 Processor -- start 2024-04-25T19:13:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.171+0000 7f7cc8d84640 1 -- start start 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc8d84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 0x7f7cc41f95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc8d84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b7650 0x7f7cc40ac530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc8d84640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cc40b9470 0x7f7cc4162070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc8d84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc40d07e0 con 0x7f7cc40b7650 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc8d84640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc40d0500 con 0x7f7cc40b9470 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc8d84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc41649c0 con 0x7f7cc40b4e30 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b7650 0x7f7cc40ac530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b7650 0x7f7cc40ac530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35780/0 (socket says 172.21.15.28:35780) 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 -- 172.21.15.28:0/2511079385 learned_addr learned my addr 172.21.15.28:0/2511079385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc259c640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 0x7f7cc41f95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc2d9d640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cc40b9470 0x7f7cc4162070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 -- 172.21.15.28:0/2511079385 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 msgr2=0x7f7cc41f95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 0x7f7cc41f95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 -- 172.21.15.28:0/2511079385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cc40b9470 msgr2=0x7f7cc4162070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cc40b9470 0x7f7cc4162070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 -- 172.21.15.28:0/2511079385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc40aca70 con 0x7f7cc40b7650 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.172+0000 7f7cc1d9b640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b7650 0x7f7cc40ac530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.173+0000 7f7cbaffd640 1 -- 172.21.15.28:0/2511079385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac037040 con 0x7f7cc40b7650 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.173+0000 7f7cbaffd640 1 -- 172.21.15.28:0/2511079385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cac042070 con 0x7f7cc40b7650 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.173+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc41e9ce0 con 0x7f7cc40b7650 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.173+0000 7f7cbaffd640 1 -- 172.21.15.28:0/2511079385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cac030bd0 con 0x7f7cc40b7650 2024-04-25T19:13:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.173+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc41ea2a0 con 0x7f7cc40b7650 2024-04-25T19:13:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.174+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc40acc40 con 0x7f7cc40b7650 2024-04-25T19:13:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.174+0000 7f7cbaffd640 1 -- 172.21.15.28:0/2511079385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7cac02ad10 con 0x7f7cc40b7650 2024-04-25T19:13:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.175+0000 7f7cbaffd640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7ca40744b0 0x7f7ca40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.175+0000 7f7cbaffd640 1 -- 172.21.15.28:0/2511079385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f7cac0b5ff0 con 0x7f7cc40b7650 2024-04-25T19:13:22.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.175+0000 7f7cc259c640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7ca40744b0 0x7f7ca40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.175+0000 7f7cc259c640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7ca40744b0 0x7f7ca40769c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40070b0 tx=0x7f7cb4007040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.180+0000 7f7cbaffd640 1 -- 172.21.15.28:0/2511079385 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7cac07fd70 con 0x7f7cc40b7650 2024-04-25T19:13:22.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.402+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7cc4050740 con 0x7f7cc40b7650 2024-04-25T19:13:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.403+0000 7f7cbaffd640 1 -- 172.21.15.28:0/2511079385 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6483 (secure 0 0 0) 0x7f7cac084070 con 0x7f7cc40b7650 2024-04-25T19:13:22.405 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:22.405 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-04-25T19:13:22.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7ca40744b0 msgr2=0x7f7ca40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7ca40744b0 0x7f7ca40769c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7cb40070b0 tx=0x7f7cb4007040 comp rx=0 tx=0).stop 2024-04-25T19:13:22.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b7650 msgr2=0x7f7cc40ac530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b7650 0x7f7cc40ac530 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac0046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 shutdown_connections 2024-04-25T19:13:22.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7ca40744b0 0x7f7ca40769c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7cc40b9470 0x7f7cc4162070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cc40b7650 0x7f7cc40ac530 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 --2- 172.21.15.28:0/2511079385 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7cc40b4e30 0x7f7cc41f95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 >> 172.21.15.28:0/2511079385 conn(0x7f7cc40d2650 msgr2=0x7f7cc40b5550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:22.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 shutdown_connections 2024-04-25T19:13:22.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.405+0000 7f7cc8d84640 1 -- 172.21.15.28:0/2511079385 wait complete. 2024-04-25T19:13:22.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:22.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.528+0000 7fb1af049640 1 Processor -- start 2024-04-25T19:13:22.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.528+0000 7fb1af049640 1 -- start start 2024-04-25T19:13:22.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.529+0000 7fb1af049640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80b7d70 0x7fb1a815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.529+0000 7fb1af049640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb1a81606f0 0x7fb1a8160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.529+0000 7fb1af049640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a81614a0 con 0x7fb1a8164990 2024-04-25T19:13:22.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.529+0000 7fb1af049640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a8163eb0 con 0x7fb1a81606f0 2024-04-25T19:13:22.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.529+0000 7fb1af049640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a8164190 con 0x7fb1a80b7d70 2024-04-25T19:13:22.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.529+0000 7fb1acde5640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb1a8164990 0x7fb1a80b7660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39810/0 (socket says 172.21.15.28:39810) 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.529+0000 7fb1acde5640 1 -- 172.21.15.28:0/353382624 learned_addr learned my addr 172.21.15.28:0/353382624 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1ad5e6640 1 --2- 172.21.15.28:0/353382624 >> v2:172.21.15.53:3300/0 conn(0x7fb1a81606f0 0x7fb1a8160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a7fff640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80b7d70 0x7fb1a815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2167508181 0 0) 0x7fb1a81614a0 con 0x7fb1a8164990 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb198003520 con 0x7fb1a8164990 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1910077208 0 0) 0x7fb198003520 con 0x7fb1a8164990 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80b7d70 msgr2=0x7fb1a815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80b7d70 0x7fb1a815dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 >> v2:172.21.15.53:3300/0 conn(0x7fb1a81606f0 msgr2=0x7fb1a8160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 --2- 172.21.15.28:0/353382624 >> v2:172.21.15.53:3300/0 conn(0x7fb1a81606f0 0x7fb1a8160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a8164470 con 0x7fb1a8164990 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fb19c001670 con 0x7fb1a8164990 2024-04-25T19:13:22.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 >> v1:172.21.15.28:6789/0 conn(0x7fb1a8164990 legacy=0x7fb1a80b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80b7d70 0x7fb1980039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb198003f30 0x7fb198004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198004d40 0x7fb198005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8163eb0 con 0x7fb198004d40 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8164190 con 0x7fb198003f30 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb19800ba10 con 0x7fb1a80b7d70 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1ad5e6640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198004d40 0x7fb198005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1acde5640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80b7d70 0x7fb1980039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.530+0000 7fb1a7fff640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb198003f30 0x7fb198004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.531+0000 7fb1ad5e6640 1 -- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80b7d70 msgr2=0x7fb1980039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.531+0000 7fb1ad5e6640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80b7d70 0x7fb1980039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.531+0000 7fb1ad5e6640 1 -- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb198003f30 msgr2=0x7fb198004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.531+0000 7fb1ad5e6640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb198003f30 0x7fb198004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.531+0000 7fb1ad5e6640 1 -- 172.21.15.28:0/353382624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb198003520 con 0x7fb198004d40 2024-04-25T19:13:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.531+0000 7fb1ad5e6640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198004d40 0x7fb198005150 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009f90 tx=0x7fb198005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb2c21284fc5268 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.531+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0009c10 con 0x7fb198004d40 2024-04-25T19:13:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a0036070 con 0x7fb198004d40 2024-04-25T19:13:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1a77fe640 1 -- 172.21.15.28:0/353382624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0031880 con 0x7fb198004d40 2024-04-25T19:13:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1af049640 1 -- 172.21.15.28:0/353382624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198004d40 msgr2=0x7fb198005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1af049640 1 --2- 172.21.15.28:0/353382624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb198004d40 0x7fb198005150 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009f90 tx=0x7fb198005830 comp rx=0 tx=0).stop 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1ad5e6640 1 -- 172.21.15.28:0/353382624 reap_dead start 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1af049640 1 -- 172.21.15.28:0/353382624 shutdown_connections 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1af049640 1 -- 172.21.15.28:0/353382624 >> 172.21.15.28:0/353382624 conn(0x7fb1a80d2840 msgr2=0x7fb1a80d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1af049640 1 -- 172.21.15.28:0/353382624 shutdown_connections 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.532+0000 7fb1af049640 1 -- 172.21.15.28:0/353382624 wait complete. 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.534+0000 7fb1af049640 1 Processor -- start 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.534+0000 7fb1af049640 1 -- start start 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.535+0000 7fb1af049640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80c9c10 0x7fb1a815abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.535+0000 7fb1af049640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a80b7890 0x7fb1a80b6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.535+0000 7fb1af049640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1a80b6ad0 0x7fb1a81f9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.535+0000 7fb1af049640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a815dcb0 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.535+0000 7fb1af049640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8161250 con 0x7fb1a80b7890 2024-04-25T19:13:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.535+0000 7fb1af049640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a815de30 con 0x7fb1a80c9c10 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1a80b6ad0 0x7fb1a81f9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1a80b6ad0 0x7fb1a81f9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35790/0 (socket says 172.21.15.28:35790) 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 -- 172.21.15.28:0/3672285002 learned_addr learned my addr 172.21.15.28:0/3672285002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1acde5640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80c9c10 0x7fb1a815abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1a7fff640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a80b7890 0x7fb1a80b6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 -- 172.21.15.28:0/3672285002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80c9c10 msgr2=0x7fb1a815abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80c9c10 0x7fb1a815abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 -- 172.21.15.28:0/3672285002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a80b7890 msgr2=0x7fb1a80b6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a80b7890 0x7fb1a80b6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 -- 172.21.15.28:0/3672285002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a80b6ee0 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1ad5e6640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1a80b6ad0 0x7fb1a81f9a10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009f90 tx=0x7fb1a0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1a57fa640 1 -- 172.21.15.28:0/3672285002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0009c10 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1a57fa640 1 -- 172.21.15.28:0/3672285002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a0003990 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a81f6880 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1a57fa640 1 -- 172.21.15.28:0/3672285002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0003d80 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.536+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a81f6e40 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.537+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a8050740 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.538+0000 7fb1a57fa640 1 -- 172.21.15.28:0/3672285002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb1a0005d10 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.539+0000 7fb1a57fa640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb18c0744b0 0x7fb18c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.539+0000 7fb1acde5640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb18c0744b0 0x7fb18c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.539+0000 7fb1a57fa640 1 -- 172.21.15.28:0/3672285002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fb1a00b6800 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.539+0000 7fb1acde5640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb18c0744b0 0x7fb18c0769c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb1a81faad0 tx=0x7fb19c0078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.544+0000 7fb1a57fa640 1 -- 172.21.15.28:0/3672285002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1a004a060 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.728+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb1a8162da0 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.730+0000 7fb1a57fa640 1 -- 172.21.15.28:0/3672285002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+18884 (secure 0 0 0) 0x7fb1a80b6ee0 con 0x7fb1a80b6ad0 2024-04-25T19:13:22.732 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:22.732 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:10.963059+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb18c0744b0 msgr2=0x7fb18c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb18c0744b0 0x7fb18c0769c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb1a81faad0 tx=0x7fb19c0078f0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1a80b6ad0 msgr2=0x7fb1a81f9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1a80b6ad0 0x7fb1a81f9a10 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009f90 tx=0x7fb1a0009750 comp rx=0 tx=0).stop 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 shutdown_connections 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb18c0744b0 0x7fb18c0769c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1a80b6ad0 0x7fb1a81f9a10 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1a80b7890 0x7fb1a80b6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 --2- 172.21.15.28:0/3672285002 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1a80c9c10 0x7fb1a815abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 >> 172.21.15.28:0/3672285002 conn(0x7fb1a80d2840 msgr2=0x7fb1a80bb410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 shutdown_connections 2024-04-25T19:13:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.732+0000 7fb1af049640 1 -- 172.21.15.28:0/3672285002 wait complete. 2024-04-25T19:13:22.745 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T19:13:22.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.855+0000 7f00c0e4a640 1 Processor -- start 2024-04-25T19:13:22.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.855+0000 7f00c0e4a640 1 -- start start 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.856+0000 7f00c0e4a640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00bc09f790 0x7f00bc06d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.856+0000 7f00c0e4a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00bc0702f0 0x7f00bc06dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.856+0000 7f00c0e4a640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00bc15d610 con 0x7f00bc0d0dd0 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.856+0000 7f00c0e4a640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f00bc15d8f0 con 0x7f00bc0702f0 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.856+0000 7f00c0e4a640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f00bc15dbd0 con 0x7f00bc09f790 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.857+0000 7f00bad9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00bc0702f0 0x7f00bc06dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.857+0000 7f00bad9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00bc0702f0 0x7f00bc06dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35800/0 (socket says 172.21.15.28:35800) 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.857+0000 7f00bad9d640 1 -- 172.21.15.28:0/3146591638 learned_addr learned my addr 172.21.15.28:0/3146591638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.857+0000 7f00ba59c640 1 --2- 172.21.15.28:0/3146591638 >> v2:172.21.15.53:3300/0 conn(0x7f00bc09f790 0x7f00bc06d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.857+0000 7f00bad9d640 1 -- 172.21.15.28:0/3146591638 >> v2:172.21.15.53:3300/0 conn(0x7f00bc09f790 msgr2=0x7f00bc06d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.857+0000 7f00bad9d640 1 --2- 172.21.15.28:0/3146591638 >> v2:172.21.15.53:3300/0 conn(0x7f00bc09f790 0x7f00bc06d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.857+0000 7f00bad9d640 1 -- 172.21.15.28:0/3146591638 >> v1:172.21.15.53:6789/0 conn(0x7f00bc0d0dd0 legacy=0x7f00bc160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.857+0000 7f00bad9d640 1 -- 172.21.15.28:0/3146591638 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00bc15deb0 con 0x7f00bc0702f0 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.858+0000 7f00bad9d640 1 --2- 172.21.15.28:0/3146591638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc06dac0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f00bc06f150 tx=0x7f00ac02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4fbf73aa0672a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.858+0000 7f00b959a640 1 -- 172.21.15.28:0/3146591638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac036070 con 0x7f00bc0702f0 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.858+0000 7f00b959a640 1 -- 172.21.15.28:0/3146591638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00ac031600 con 0x7f00bc0702f0 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.858+0000 7f00b959a640 1 -- 172.21.15.28:0/3146591638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac0319d0 con 0x7f00bc0702f0 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.859+0000 7f00c0e4a640 1 -- 172.21.15.28:0/3146591638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 msgr2=0x7f00bc06dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.859+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/3146591638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc06dac0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f00bc06f150 tx=0x7f00ac02a730 comp rx=0 tx=0).stop 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.859+0000 7f00c0e4a640 1 -- 172.21.15.28:0/3146591638 shutdown_connections 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.859+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/3146591638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc06dac0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.859+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/3146591638 >> v2:172.21.15.53:3300/0 conn(0x7f00bc09f790 0x7f00bc06d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.859+0000 7f00c0e4a640 1 -- 172.21.15.28:0/3146591638 >> 172.21.15.28:0/3146591638 conn(0x7f00bc0732b0 msgr2=0x7f00bc0736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:22.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.859+0000 7f00c0e4a640 1 -- 172.21.15.28:0/3146591638 shutdown_connections 2024-04-25T19:13:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.859+0000 7f00c0e4a640 1 -- 172.21.15.28:0/3146591638 wait complete. 2024-04-25T19:13:22.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.861+0000 7f00c0e4a640 1 Processor -- start 2024-04-25T19:13:22.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.861+0000 7f00c0e4a640 1 -- start start 2024-04-25T19:13:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00c0e4a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc1f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00c0e4a640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc09f790 0x7f00bc1fa5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00c0e4a640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc0d0dd0 0x7f00bc0cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00c0e4a640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc15ad80 con 0x7f00bc0702f0 2024-04-25T19:13:22.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00c0e4a640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f00bc0d1490 con 0x7f00bc09f790 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00c0e4a640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f00bc159590 con 0x7f00bc0d0dd0 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc1f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00ba59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc1f5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35802/0 (socket says 172.21.15.28:35802) 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00ba59c640 1 -- 172.21.15.28:0/1913695991 learned_addr learned my addr 172.21.15.28:0/1913695991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00b9d9b640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc09f790 0x7f00bc1fa5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.862+0000 7f00bad9d640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc0d0dd0 0x7f00bc0cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00ba59c640 1 -- 172.21.15.28:0/1913695991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc0d0dd0 msgr2=0x7f00bc0cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00ba59c640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc0d0dd0 0x7f00bc0cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00ba59c640 1 -- 172.21.15.28:0/1913695991 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc09f790 msgr2=0x7f00bc1fa5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00ba59c640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc09f790 0x7f00bc1fa5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00ba59c640 1 -- 172.21.15.28:0/1913695991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00bc158070 con 0x7f00bc0702f0 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00ba59c640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc1f5f60 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f00b400e8a0 tx=0x7f00b400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00aaffd640 1 -- 172.21.15.28:0/1913695991 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b4017070 con 0x7f00bc0702f0 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00aaffd640 1 -- 172.21.15.28:0/1913695991 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00b400ee90 con 0x7f00bc0702f0 2024-04-25T19:13:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00aaffd640 1 -- 172.21.15.28:0/1913695991 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00b400cae0 con 0x7f00bc0702f0 2024-04-25T19:13:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00bc158350 con 0x7f00bc0702f0 2024-04-25T19:13:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.863+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00bc1f9dd0 con 0x7f00bc0702f0 2024-04-25T19:13:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.864+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00bc050740 con 0x7f00bc0702f0 2024-04-25T19:13:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.865+0000 7f00aaffd640 1 -- 172.21.15.28:0/1913695991 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f00b401a070 con 0x7f00bc0702f0 2024-04-25T19:13:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.865+0000 7f00aaffd640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f009c0744e0 0x7f009c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:22.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.865+0000 7f00aaffd640 1 -- 172.21.15.28:0/1913695991 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f00b4095690 con 0x7f00bc0702f0 2024-04-25T19:13:22.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.865+0000 7f00b9d9b640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f009c0744e0 0x7f009c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:22.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.866+0000 7f00b9d9b640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f009c0744e0 0x7f009c0769f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f00bc15f750 tx=0x7f00b0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:22.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:22.870+0000 7f00aaffd640 1 -- 172.21.15.28:0/1913695991 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00b405f400 con 0x7f00bc0702f0 2024-04-25T19:13:23.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.097+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00bc074220 con 0x7f00bc0702f0 2024-04-25T19:13:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.098+0000 7f00aaffd640 1 -- 172.21.15.28:0/1913695991 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6483 (secure 0 0 0) 0x7f00b4063700 con 0x7f00bc0702f0 2024-04-25T19:13:23.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:23.100 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-04-25T19:13:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f009c0744e0 msgr2=0x7f009c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f009c0744e0 0x7f009c0769f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f00bc15f750 tx=0x7f00b0009040 comp rx=0 tx=0).stop 2024-04-25T19:13:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 msgr2=0x7f00bc1f5f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc1f5f60 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f00b400e8a0 tx=0x7f00b400ed60 comp rx=0 tx=0).stop 2024-04-25T19:13:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 shutdown_connections 2024-04-25T19:13:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f009c0744e0 0x7f009c0769f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f00bc0d0dd0 0x7f00bc0cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00bc09f790 0x7f00bc1fa5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 --2- 172.21.15.28:0/1913695991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00bc0702f0 0x7f00bc1f5f60 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 >> 172.21.15.28:0/1913695991 conn(0x7f00bc0732b0 msgr2=0x7f00bc157560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 shutdown_connections 2024-04-25T19:13:23.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.100+0000 7f00c0e4a640 1 -- 172.21.15.28:0/1913695991 wait complete. 2024-04-25T19:13:23.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.223+0000 7fa8e34b6640 1 Processor -- start 2024-04-25T19:13:23.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.223+0000 7fa8e34b6640 1 -- start start 2024-04-25T19:13:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.224+0000 7fa8e34b6640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc0b7d70 0x7fa8dc15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.224+0000 7fa8e34b6640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8dc1606f0 0x7fa8dc160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.224+0000 7fa8e34b6640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8dc1614a0 con 0x7fa8dc164990 2024-04-25T19:13:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.224+0000 7fa8e34b6640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fa8dc163eb0 con 0x7fa8dc1606f0 2024-04-25T19:13:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.224+0000 7fa8e34b6640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa8dc164190 con 0x7fa8dc0b7d70 2024-04-25T19:13:23.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.224+0000 7fa8e1252640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa8dc164990 0x7fa8dc0b7660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39824/0 (socket says 172.21.15.28:39824) 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.224+0000 7fa8e1a53640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8dc1606f0 0x7fa8dc160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.224+0000 7fa8e1252640 1 -- 172.21.15.28:0/1245195741 learned_addr learned my addr 172.21.15.28:0/1245195741 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.225+0000 7fa8e0a51640 1 --2- 172.21.15.28:0/1245195741 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc0b7d70 0x7fa8dc15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.225+0000 7fa8e1a53640 1 -- 172.21.15.28:0/1245195741 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc0b7d70 msgr2=0x7fa8dc15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.225+0000 7fa8e1a53640 1 --2- 172.21.15.28:0/1245195741 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc0b7d70 0x7fa8dc15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.225+0000 7fa8e1a53640 1 -- 172.21.15.28:0/1245195741 >> v1:172.21.15.28:6789/0 conn(0x7fa8dc164990 legacy=0x7fa8dc0b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.225+0000 7fa8d3fff640 1 -- 172.21.15.28:0/1245195741 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599937182 0 0) 0x7fa8dc1614a0 con 0x7fa8dc164990 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.225+0000 7fa8e1a53640 1 -- 172.21.15.28:0/1245195741 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc164470 con 0x7fa8dc1606f0 2024-04-25T19:13:23.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.225+0000 7fa8e1a53640 1 --2- 172.21.15.28:0/1245195741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc1606f0 0x7fa8dc160b00 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009d90 tx=0x7fa8d402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af63a8f4b26c67a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8d3fff640 1 -- 172.21.15.28:0/1245195741 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4007740 con 0x7fa8dc1606f0 2024-04-25T19:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8d3fff640 1 -- 172.21.15.28:0/1245195741 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d40078e0 con 0x7fa8dc1606f0 2024-04-25T19:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8d3fff640 1 -- 172.21.15.28:0/1245195741 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4007cb0 con 0x7fa8dc1606f0 2024-04-25T19:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8e34b6640 1 -- 172.21.15.28:0/1245195741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc1606f0 msgr2=0x7fa8dc160b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/1245195741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc1606f0 0x7fa8dc160b00 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009d90 tx=0x7fa8d402a660 comp rx=0 tx=0).stop 2024-04-25T19:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8e34b6640 1 -- 172.21.15.28:0/1245195741 shutdown_connections 2024-04-25T19:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/1245195741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc1606f0 0x7fa8dc160b00 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/1245195741 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc0b7d70 0x7fa8dc15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8e34b6640 1 -- 172.21.15.28:0/1245195741 >> 172.21.15.28:0/1245195741 conn(0x7fa8dc0d2840 msgr2=0x7fa8dc0d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8e34b6640 1 -- 172.21.15.28:0/1245195741 shutdown_connections 2024-04-25T19:13:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.226+0000 7fa8e34b6640 1 -- 172.21.15.28:0/1245195741 wait complete. 2024-04-25T19:13:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.228+0000 7fa8e34b6640 1 Processor -- start 2024-04-25T19:13:23.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.228+0000 7fa8e34b6640 1 -- start start 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.229+0000 7fa8e34b6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8dc0b7d70 0x7fa8dc1e9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.229+0000 7fa8e34b6640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc1606f0 0x7fa8dc14fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.229+0000 7fa8e34b6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc164990 0x7fa8dc15ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.229+0000 7fa8e34b6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc0b45d0 con 0x7fa8dc164990 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.229+0000 7fa8e34b6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa8dc161250 con 0x7fa8dc0b7d70 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.229+0000 7fa8e34b6640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa8dc0b4750 con 0x7fa8dc1606f0 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc164990 0x7fa8dc15ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc164990 0x7fa8dc15ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35822/0 (socket says 172.21.15.28:35822) 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 -- 172.21.15.28:0/3260169939 learned_addr learned my addr 172.21.15.28:0/3260169939 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1252640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8dc0b7d70 0x7fa8dc1e9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e0a51640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc1606f0 0x7fa8dc14fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 -- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc1606f0 msgr2=0x7fa8dc14fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc1606f0 0x7fa8dc14fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 -- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8dc0b7d70 msgr2=0x7fa8dc1e9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8dc0b7d70 0x7fa8dc1e9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 -- 172.21.15.28:0/3260169939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8dc0ce990 con 0x7fa8dc164990 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1252640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8dc0b7d70 0x7fa8dc1e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e0a51640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc1606f0 0x7fa8dc14fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e1a53640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc164990 0x7fa8dc15ed00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009d90 tx=0x7fa8d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8d1ffb640 1 -- 172.21.15.28:0/3260169939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d403f070 con 0x7fa8dc164990 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8d1ffb640 1 -- 172.21.15.28:0/3260169939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8d4031bc0 con 0x7fa8dc164990 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8dc0cec70 con 0x7fa8dc164990 2024-04-25T19:13:23.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.230+0000 7fa8d1ffb640 1 -- 172.21.15.28:0/3260169939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4043410 con 0x7fa8dc164990 2024-04-25T19:13:23.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.231+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8dc0cf230 con 0x7fa8dc164990 2024-04-25T19:13:23.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.231+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8dc0ce700 con 0x7fa8dc164990 2024-04-25T19:13:23.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.232+0000 7fa8d1ffb640 1 -- 172.21.15.28:0/3260169939 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa8d4007790 con 0x7fa8dc164990 2024-04-25T19:13:23.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.233+0000 7fa8d1ffb640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa8c00744e0 0x7fa8c00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.233+0000 7fa8d1ffb640 1 -- 172.21.15.28:0/3260169939 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fa8d40b6af0 con 0x7fa8dc164990 2024-04-25T19:13:23.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.233+0000 7fa8e1252640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa8c00744e0 0x7fa8c00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.233+0000 7fa8e1252640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa8c00744e0 0x7fa8c00769f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8000f60 tx=0x7fa8d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.237+0000 7fa8d1ffb640 1 -- 172.21.15.28:0/3260169939 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8d4080ec0 con 0x7fa8dc164990 2024-04-25T19:13:23.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.461+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8dc050740 con 0x7fa8dc164990 2024-04-25T19:13:23.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.461+0000 7fa8d1ffb640 1 -- 172.21.15.28:0/3260169939 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+6483 (secure 0 0 0) 0x7fa8d40807c0 con 0x7fa8dc164990 2024-04-25T19:13:23.463 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:23.463 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":19,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:12:44.063277+0000","modified":"2024-04-25T19:12:54.689116+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2},{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:23.464 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 30 2024-04-25T19:13:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa8c00744e0 msgr2=0x7fa8c00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa8c00744e0 0x7fa8c00769f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8000f60 tx=0x7fa8d8009040 comp rx=0 tx=0).stop 2024-04-25T19:13:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc164990 msgr2=0x7fa8dc15ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc164990 0x7fa8dc15ed00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fa8d4009d90 tx=0x7fa8d4004a40 comp rx=0 tx=0).stop 2024-04-25T19:13:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 shutdown_connections 2024-04-25T19:13:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa8c00744e0 0x7fa8c00769f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8dc164990 0x7fa8dc15ed00 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa8dc1606f0 0x7fa8dc14fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 --2- 172.21.15.28:0/3260169939 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8dc0b7d70 0x7fa8dc1e9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.463+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 >> 172.21.15.28:0/3260169939 conn(0x7fa8dc0d2840 msgr2=0x7fa8dc0d44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:23.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.464+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 shutdown_connections 2024-04-25T19:13:23.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.464+0000 7fa8e34b6640 1 -- 172.21.15.28:0/3260169939 wait complete. 2024-04-25T19:13:23.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.586+0000 7fdcbcc6e640 1 Processor -- start 2024-04-25T19:13:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.586+0000 7fdcbcc6e640 1 -- start start 2024-04-25T19:13:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcbcc6e640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcb80b5130 0x7fdcb8161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcbcc6e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb815f7d0 0x7fdcb815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcbcc6e640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb8163e80 con 0x7fdcb806ded0 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcbcc6e640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fdcb8163c30 con 0x7fdcb80b5130 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcbcc6e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb8164a90 con 0x7fdcb815f7d0 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcb659c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcb80b5130 0x7fdcb8161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcb659c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcb80b5130 0x7fdcb8161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35838/0 (socket says 172.21.15.28:35838) 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcb659c640 1 -- 172.21.15.28:0/2579512419 learned_addr learned my addr 172.21.15.28:0/2579512419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcb6d9d640 1 --2- 172.21.15.28:0/2579512419 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb815f7d0 0x7fdcb815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcb659c640 1 -- 172.21.15.28:0/2579512419 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb815f7d0 msgr2=0x7fdcb815fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcb659c640 1 --2- 172.21.15.28:0/2579512419 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb815f7d0 0x7fdcb815fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcb659c640 1 -- 172.21.15.28:0/2579512419 >> v1:172.21.15.53:6789/0 conn(0x7fdcb806ded0 legacy=0x7fdcb806e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:23.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.588+0000 7fdcb659c640 1 -- 172.21.15.28:0/2579512419 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb8164160 con 0x7fdcb80b5130 2024-04-25T19:13:23.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.589+0000 7fdcb659c640 1 --2- 172.21.15.28:0/2579512419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb80b5130 0x7fdcb8161130 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdcb806ec40 tx=0x7fdcb002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8d6b9ac8f2804e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.589+0000 7fdcb559a640 1 -- 172.21.15.28:0/2579512419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0037070 con 0x7fdcb80b5130 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.589+0000 7fdcb559a640 1 -- 172.21.15.28:0/2579512419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcb0032440 con 0x7fdcb80b5130 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.589+0000 7fdcb559a640 1 -- 172.21.15.28:0/2579512419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcb0032810 con 0x7fdcb80b5130 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.590+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2579512419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb80b5130 msgr2=0x7fdcb8161130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.590+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2579512419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb80b5130 0x7fdcb8161130 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fdcb806ec40 tx=0x7fdcb002a1e0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.590+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2579512419 shutdown_connections 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.590+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2579512419 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb815f7d0 0x7fdcb815fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.590+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2579512419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb80b5130 0x7fdcb8161130 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.590+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2579512419 >> 172.21.15.28:0/2579512419 conn(0x7fdcb80b7ad0 msgr2=0x7fdcb80b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.590+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2579512419 shutdown_connections 2024-04-25T19:13:23.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.590+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2579512419 wait complete. 2024-04-25T19:13:23.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.592+0000 7fdcbcc6e640 1 Processor -- start 2024-04-25T19:13:23.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.592+0000 7fdcbcc6e640 1 -- start start 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.593+0000 7fdcbcc6e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb806ded0 0x7fdcb81f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.593+0000 7fdcbcc6e640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcb80b5130 0x7fdcb81f9b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.593+0000 7fdcbcc6e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb815f7d0 0x7fdcb81fa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.593+0000 7fdcbcc6e640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb8160120 con 0x7fdcb815f7d0 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.593+0000 7fdcbcc6e640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb8164720 con 0x7fdcb80b5130 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.593+0000 7fdcbcc6e640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb81602a0 con 0x7fdcb806ded0 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.593+0000 7fdcb6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb815f7d0 0x7fdcb81fa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb6d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb815f7d0 0x7fdcb81fa0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35852/0 (socket says 172.21.15.28:35852) 2024-04-25T19:13:23.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb6d9d640 1 -- 172.21.15.28:0/2572188448 learned_addr learned my addr 172.21.15.28:0/2572188448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb659c640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb806ded0 0x7fdcb81f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb5d9b640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcb80b5130 0x7fdcb81f9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb6d9d640 1 -- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb806ded0 msgr2=0x7fdcb81f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb6d9d640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb806ded0 0x7fdcb81f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb6d9d640 1 -- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcb80b5130 msgr2=0x7fdcb81f9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb6d9d640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcb80b5130 0x7fdcb81f9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb6d9d640 1 -- 172.21.15.28:0/2572188448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb81ec3c0 con 0x7fdcb815f7d0 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb659c640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb806ded0 0x7fdcb81f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb5d9b640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcb80b5130 0x7fdcb81f9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcb6d9d640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb815f7d0 0x7fdcb81fa0d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fdca800cc00 tx=0x7fdca800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcaeffd640 1 -- 172.21.15.28:0/2572188448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca800ce10 con 0x7fdcb815f7d0 2024-04-25T19:13:23.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcaeffd640 1 -- 172.21.15.28:0/2572188448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca8003990 con 0x7fdcb815f7d0 2024-04-25T19:13:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb80ac5b0 con 0x7fdcb815f7d0 2024-04-25T19:13:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcaeffd640 1 -- 172.21.15.28:0/2572188448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca8003d60 con 0x7fdcb815f7d0 2024-04-25T19:13:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.594+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb80acb70 con 0x7fdcb815f7d0 2024-04-25T19:13:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.595+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb8050740 con 0x7fdcb815f7d0 2024-04-25T19:13:23.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.596+0000 7fdcaeffd640 1 -- 172.21.15.28:0/2572188448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdca8003f40 con 0x7fdcb815f7d0 2024-04-25T19:13:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.596+0000 7fdcaeffd640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdc980744b0 0x7fdc980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.597+0000 7fdcaeffd640 1 -- 172.21.15.28:0/2572188448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fdca8095960 con 0x7fdcb815f7d0 2024-04-25T19:13:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.597+0000 7fdcb659c640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdc980744b0 0x7fdc980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.597+0000 7fdcb659c640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdc980744b0 0x7fdc980769c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0034000 tx=0x7fdcb002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.601+0000 7fdcaeffd640 1 -- 172.21.15.28:0/2572188448 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdca8014030 con 0x7fdcb815f7d0 2024-04-25T19:13:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.793+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdcb80d3bf0 con 0x7fdcb815f7d0 2024-04-25T19:13:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.794+0000 7fdcaeffd640 1 -- 172.21.15.28:0/2572188448 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+18884 (secure 0 0 0) 0x7fdca805fe50 con 0x7fdcb815f7d0 2024-04-25T19:13:23.796 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:23.797 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:10.963059+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T19:12:41.809696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdc980744b0 msgr2=0x7fdc980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdc980744b0 0x7fdc980769c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fdcb0034000 tx=0x7fdcb002e000 comp rx=0 tx=0).stop 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb815f7d0 msgr2=0x7fdcb81fa0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb815f7d0 0x7fdcb81fa0d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fdca800cc00 tx=0x7fdca800a5c0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 shutdown_connections 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdc980744b0 0x7fdc980769c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb815f7d0 0x7fdcb81fa0d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdcb80b5130 0x7fdcb81f9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 --2- 172.21.15.28:0/2572188448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdcb806ded0 0x7fdcb81f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 >> 172.21.15.28:0/2572188448 conn(0x7fdcb80b7ad0 msgr2=0x7fdcb8163140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 shutdown_connections 2024-04-25T19:13:23.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.797+0000 7fdcbcc6e640 1 -- 172.21.15.28:0/2572188448 wait complete. 2024-04-25T19:13:23.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T19:13:23.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.921+0000 7fd2287a8640 1 Processor -- start 2024-04-25T19:13:23.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.921+0000 7fd2287a8640 1 -- start start 2024-04-25T19:13:23.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.922+0000 7fd2287a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 0x7fd22015dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.922+0000 7fd2287a8640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2201606f0 0x7fd220160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.922+0000 7fd2287a8640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2201614a0 con 0x7fd220164990 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.922+0000 7fd2287a8640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fd220163eb0 con 0x7fd2201606f0 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.922+0000 7fd2287a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd220164190 con 0x7fd2200b7d70 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.922+0000 7fd226d45640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2201606f0 0x7fd220160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.922+0000 7fd226d45640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2201606f0 0x7fd220160b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35856/0 (socket says 172.21.15.28:35856) 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.922+0000 7fd226d45640 1 -- 172.21.15.28:0/1996319643 learned_addr learned my addr 172.21.15.28:0/1996319643 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.923+0000 7fd225d43640 1 --2- 172.21.15.28:0/1996319643 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 0x7fd22015dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.923+0000 7fd226d45640 1 -- 172.21.15.28:0/1996319643 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 msgr2=0x7fd22015dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.923+0000 7fd226d45640 1 --2- 172.21.15.28:0/1996319643 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 0x7fd22015dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.923+0000 7fd226d45640 1 -- 172.21.15.28:0/1996319643 >> v1:172.21.15.53:6789/0 conn(0x7fd220164990 legacy=0x7fd2200b7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.923+0000 7fd226d45640 1 -- 172.21.15.28:0/1996319643 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd220164470 con 0x7fd2201606f0 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.923+0000 7fd226d45640 1 --2- 172.21.15.28:0/1996319643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd220160b00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd218009e50 tx=0x7fd21802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56de0dfcbd5a653 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd225542640 1 -- 172.21.15.28:0/1996319643 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd218036070 con 0x7fd2201606f0 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd225542640 1 -- 172.21.15.28:0/1996319643 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd218031600 con 0x7fd2201606f0 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd225542640 1 -- 172.21.15.28:0/1996319643 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2180319d0 con 0x7fd2201606f0 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd2287a8640 1 -- 172.21.15.28:0/1996319643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 msgr2=0x7fd220160b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1996319643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd220160b00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd218009e50 tx=0x7fd21802a710 comp rx=0 tx=0).stop 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd2287a8640 1 -- 172.21.15.28:0/1996319643 shutdown_connections 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1996319643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd220160b00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1996319643 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 0x7fd22015dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd2287a8640 1 -- 172.21.15.28:0/1996319643 >> 172.21.15.28:0/1996319643 conn(0x7fd2200d2840 msgr2=0x7fd2200d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.924+0000 7fd2287a8640 1 -- 172.21.15.28:0/1996319643 shutdown_connections 2024-04-25T19:13:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.925+0000 7fd2287a8640 1 -- 172.21.15.28:0/1996319643 wait complete. 2024-04-25T19:13:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.926+0000 7fd2287a8640 1 Processor -- start 2024-04-25T19:13:23.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.926+0000 7fd2287a8640 1 -- start start 2024-04-25T19:13:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.927+0000 7fd2287a8640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 0x7fd22015d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.927+0000 7fd2287a8640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd2201e9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.927+0000 7fd2287a8640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd220164990 0x7fd2201eb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd2287a8640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2200b4750 con 0x7fd2201606f0 2024-04-25T19:13:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd2287a8640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd2200b45d0 con 0x7fd220164990 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd2287a8640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd2200b48d0 con 0x7fd2200b7d70 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd2201e9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd2201e9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35862/0 (socket says 172.21.15.28:35862) 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 -- 172.21.15.28:0/1640788011 learned_addr learned my addr 172.21.15.28:0/1640788011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd226d45640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd220164990 0x7fd2201eb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd226544640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 0x7fd22015d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 -- 172.21.15.28:0/1640788011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 msgr2=0x7fd22015d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 0x7fd22015d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 -- 172.21.15.28:0/1640788011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd220164990 msgr2=0x7fd2201eb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd220164990 0x7fd2201eb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 -- 172.21.15.28:0/1640788011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2201ea3b0 con 0x7fd2201606f0 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd225d43640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd2201e9ab0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd21000d600 tx=0x7fd21000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd216ffd640 1 -- 172.21.15.28:0/1640788011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2100125f0 con 0x7fd2201606f0 2024-04-25T19:13:23.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd216ffd640 1 -- 172.21.15.28:0/1640788011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd210019070 con 0x7fd2201606f0 2024-04-25T19:13:23.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd216ffd640 1 -- 172.21.15.28:0/1640788011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2100129f0 con 0x7fd2201606f0 2024-04-25T19:13:23.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2201ea690 con 0x7fd2201606f0 2024-04-25T19:13:23.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.928+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2201eac50 con 0x7fd2201606f0 2024-04-25T19:13:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.929+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2200b9890 con 0x7fd2201606f0 2024-04-25T19:13:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.930+0000 7fd216ffd640 1 -- 172.21.15.28:0/1640788011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd210012b90 con 0x7fd2201606f0 2024-04-25T19:13:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.931+0000 7fd216ffd640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2040744e0 0x7fd2040769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.931+0000 7fd226544640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2040744e0 0x7fd2040769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.931+0000 7fd216ffd640 1 -- 172.21.15.28:0/1640788011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7fd2100957b0 con 0x7fd2201606f0 2024-04-25T19:13:23.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.931+0000 7fd226544640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2040744e0 0x7fd2040769f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd2200d5bb0 tx=0x7fd21c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:23.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:23.936+0000 7fd216ffd640 1 -- 172.21.15.28:0/1640788011 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd2200b9890 con 0x7fd2201606f0 2024-04-25T19:13:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:24.166+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fd220050740 con 0x7fd2201606f0 2024-04-25T19:13:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.018+0000 7fd216ffd640 1 -- 172.21.15.28:0/1640788011 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) ==== 161+0+0 (secure 0 0 0) 0x7fd21005f5c0 con 0x7fd2201606f0 2024-04-25T19:13:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T19:13:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2040744e0 msgr2=0x7fd2040769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2040744e0 0x7fd2040769f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd2200d5bb0 tx=0x7fd21c009040 comp rx=0 tx=0).stop 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 msgr2=0x7fd2201e9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd2201e9ab0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd21000d600 tx=0x7fd21000dac0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 shutdown_connections 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2040744e0 0x7fd2040769f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd220164990 0x7fd2201eb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2201606f0 0x7fd2201e9ab0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 --2- 172.21.15.28:0/1640788011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd2200b7d70 0x7fd22015d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 >> 172.21.15.28:0/1640788011 conn(0x7fd2200d2840 msgr2=0x7fd220162350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.021+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 shutdown_connections 2024-04-25T19:13:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.022+0000 7fd2287a8640 1 -- 172.21.15.28:0/1640788011 wait complete. 2024-04-25T19:13:25.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T19:13:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.157+0000 7f52d13ff640 1 Processor -- start 2024-04-25T19:13:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.157+0000 7f52d13ff640 1 -- start start 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52d13ff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52cc1613d0 0x7f52cc06e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52d13ff640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52cc0b7cc0 con 0x7f52cc160fc0 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52d13ff640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52cc0b7fa0 con 0x7f52cc0d45e0 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52d13ff640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc160b30 con 0x7f52cc1613d0 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52c3fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f52cc160fc0 0x7f52cc0b4a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39840/0 (socket says 172.21.15.28:39840) 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52c3fff640 1 -- 172.21.15.28:0/1547254034 learned_addr learned my addr 172.21.15.28:0/1547254034 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808036763 0 0) 0x7f52cc0b7cc0 con 0x7f52cc160fc0 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52b8003520 con 0x7f52cc160fc0 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52cb7fe640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52cc1613d0 0x7f52cc06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3885848392 0 0) 0x7f52b8003520 con 0x7f52cc160fc0 2024-04-25T19:13:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52cc1613d0 msgr2=0x7f52cc06e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52cc1613d0 0x7f52cc06e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 >> v1:172.21.15.53:6789/0 conn(0x7f52cc0d45e0 legacy=0x7f52cc0b74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc0b8280 con 0x7f52cc160fc0 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f52b4001670 con 0x7f52cc160fc0 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 >> v1:172.21.15.28:6789/0 conn(0x7f52cc160fc0 legacy=0x7f52cc0b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.159+0000 7f52ca7fc640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52cc1613d0 0x7f52b80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52ca7fc640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b8003f60 0x7f52b8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52ca7fc640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52b8004d40 0x7f52b8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc160b30 con 0x7f52b8003f60 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f52b800bb50 con 0x7f52b8004d40 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f52b800bd50 con 0x7f52cc1613d0 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52c3fff640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b8003f60 0x7f52b8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52caffd640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52cc1613d0 0x7f52b80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52cb7fe640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52b8004d40 0x7f52b8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52c3fff640 1 -- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52cc1613d0 msgr2=0x7f52b80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52c3fff640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52cc1613d0 0x7f52b80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52c3fff640 1 -- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52b8004d40 msgr2=0x7f52b8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52c3fff640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52b8004d40 0x7f52b8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.160+0000 7f52c3fff640 1 -- 172.21.15.28:0/1547254034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b8003520 con 0x7f52b8003f60 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.161+0000 7f52c3fff640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b8003f60 0x7f52b8004800 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c240 tx=0x7f52b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3b3e00f9dcdb16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.161+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4007cf0 con 0x7f52b8003f60 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.161+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b40058d0 con 0x7f52b8003f60 2024-04-25T19:13:25.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.161+0000 7f52ca7fc640 1 -- 172.21.15.28:0/1547254034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b4005ca0 con 0x7f52b8003f60 2024-04-25T19:13:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.161+0000 7f52d13ff640 1 -- 172.21.15.28:0/1547254034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b8003f60 msgr2=0x7f52b8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.161+0000 7f52d13ff640 1 --2- 172.21.15.28:0/1547254034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52b8003f60 0x7f52b8004800 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f52b400c240 tx=0x7f52b8005830 comp rx=0 tx=0).stop 2024-04-25T19:13:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.161+0000 7f52cb7fe640 1 -- 172.21.15.28:0/1547254034 reap_dead start 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.162+0000 7f52d13ff640 1 -- 172.21.15.28:0/1547254034 shutdown_connections 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.162+0000 7f52d13ff640 1 -- 172.21.15.28:0/1547254034 >> 172.21.15.28:0/1547254034 conn(0x7f52cc06d120 msgr2=0x7f52cc0d24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.162+0000 7f52d13ff640 1 -- 172.21.15.28:0/1547254034 shutdown_connections 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.162+0000 7f52d13ff640 1 -- 172.21.15.28:0/1547254034 wait complete. 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.163+0000 7f52d13ff640 1 Processor -- start 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.163+0000 7f52d13ff640 1 -- start start 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52d13ff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52b8003f60 0x7f52cc0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52d13ff640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc15a230 0x7f52cc155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52d13ff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52cc156100 0x7f52cc156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52d13ff640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc06e390 con 0x7f52cc156100 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52d13ff640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f52cc06e210 con 0x7f52cc15a230 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52d13ff640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f52cc06e510 con 0x7f52b8003f60 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52cc156100 0x7f52cc156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52cc156100 0x7f52cc156510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35878/0 (socket says 172.21.15.28:35878) 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 -- 172.21.15.28:0/2839148789 learned_addr learned my addr 172.21.15.28:0/2839148789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52c3fff640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc15a230 0x7f52cc155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52caffd640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52b8003f60 0x7f52cc0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 -- 172.21.15.28:0/2839148789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52b8003f60 msgr2=0x7f52cc0cf690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52b8003f60 0x7f52cc0cf690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 -- 172.21.15.28:0/2839148789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc15a230 msgr2=0x7f52cc155070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc15a230 0x7f52cc155070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 -- 172.21.15.28:0/2839148789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52cc156a50 con 0x7f52cc156100 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52cb7fe640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52cc156100 0x7f52cc156510 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0d3550 tx=0x7f52c400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.165+0000 7f52c37fe640 1 -- 172.21.15.28:0/2839148789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c400fde0 con 0x7f52cc156100 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.166+0000 7f52c37fe640 1 -- 172.21.15.28:0/2839148789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c401d070 con 0x7f52cc156100 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.166+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52cc0ccbb0 con 0x7f52cc156100 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.166+0000 7f52c37fe640 1 -- 172.21.15.28:0/2839148789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c4015670 con 0x7f52cc156100 2024-04-25T19:13:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.166+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52cc0cd170 con 0x7f52cc156100 2024-04-25T19:13:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.167+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52cc050740 con 0x7f52cc156100 2024-04-25T19:13:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.167+0000 7f52c37fe640 1 -- 172.21.15.28:0/2839148789 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f52c4015810 con 0x7f52cc156100 2024-04-25T19:13:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.168+0000 7f52c37fe640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ac0744e0 0x7f52ac0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.168+0000 7f52c37fe640 1 -- 172.21.15.28:0/2839148789 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f52c4095d40 con 0x7f52cc156100 2024-04-25T19:13:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.168+0000 7f52caffd640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ac0744e0 0x7f52ac0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.168+0000 7f52caffd640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ac0744e0 0x7f52ac0769f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52cc08e410 tx=0x7f52bc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.172+0000 7f52c37fe640 1 -- 172.21.15.28:0/2839148789 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52cc156a50 con 0x7f52cc156100 2024-04-25T19:13:25.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:25.398+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f52cc1622d0 con 0x7f52cc156100 2024-04-25T19:13:26.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.020+0000 7f52c37fe640 1 -- 172.21.15.28:0/2839148789 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v32) ==== 104+0+0 (secure 0 0 0) 0x7f52c4010040 con 0x7f52cc156100 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ac0744e0 msgr2=0x7f52ac0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ac0744e0 0x7f52ac0769f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f52cc08e410 tx=0x7f52bc009040 comp rx=0 tx=0).stop 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52cc156100 msgr2=0x7f52cc156510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52cc156100 0x7f52cc156510 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0d3550 tx=0x7f52c400f580 comp rx=0 tx=0).stop 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 shutdown_connections 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f52ac0744e0 0x7f52ac0769f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52cc156100 0x7f52cc156510 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f52cc15a230 0x7f52cc155070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 --2- 172.21.15.28:0/2839148789 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f52b8003f60 0x7f52cc0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 >> 172.21.15.28:0/2839148789 conn(0x7f52cc06d120 msgr2=0x7f52cc0b8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 shutdown_connections 2024-04-25T19:13:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.023+0000 7f52d13ff640 1 -- 172.21.15.28:0/2839148789 wait complete. 2024-04-25T19:13:26.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T19:13:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.160+0000 7f25c4cf0640 1 Processor -- start 2024-04-25T19:13:26.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.160+0000 7f25c4cf0640 1 -- start start 2024-04-25T19:13:26.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.161+0000 7f25c4cf0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c00ce980 0x7f25c00ced90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.161+0000 7f25c4cf0640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c00cdd10 con 0x7f25c00d1e40 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25c4cf0640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c006de00 con 0x7f25c00b9210 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25c4cf0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f25c009cf80 con 0x7f25c00ce980 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f25c00d1e40 0x7f25c00d2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39854/0 (socket says 172.21.15.28:39854) 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bed9d640 1 -- 172.21.15.28:0/3184545670 learned_addr learned my addr 172.21.15.28:0/3184545670 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bdd9b640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c00ce980 0x7f25c00ced90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 154907185 0 0) 0x7f25c00cdd10 con 0x7f25c00d1e40 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25a8003520 con 0x7f25c00d1e40 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1097543565 0 0) 0x7f25a8003520 con 0x7f25c00d1e40 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c00ce980 msgr2=0x7f25c00ced90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c00ce980 0x7f25c00ced90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 >> v1:172.21.15.53:6789/0 conn(0x7f25c00b9210 legacy=0x7f25c006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c006d940 con 0x7f25c00d1e40 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f25b00019f0 con 0x7f25c00d1e40 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 >> v1:172.21.15.28:6789/0 conn(0x7f25c00d1e40 legacy=0x7f25c00d2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c00ce980 0x7f25a80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8003f40 0x7f25a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25a8004d40 0x7f25a8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25a800bb50 con 0x7f25a8004d40 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f25a800bd00 con 0x7f25a8003f40 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.162+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f25a800bf00 con 0x7f25c00ce980 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.163+0000 7f25bed9d640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25a8004d40 0x7f25a8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.163+0000 7f25be59c640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c00ce980 0x7f25a80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.163+0000 7f25bdd9b640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8003f40 0x7f25a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.163+0000 7f25bed9d640 1 -- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c00ce980 msgr2=0x7f25a80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.163+0000 7f25bed9d640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c00ce980 0x7f25a80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.163+0000 7f25bed9d640 1 -- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8003f40 msgr2=0x7f25a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:26.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.163+0000 7f25bed9d640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8003f40 0x7f25a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.163+0000 7f25bed9d640 1 -- 172.21.15.28:0/3184545670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a8003520 con 0x7f25a8004d40 2024-04-25T19:13:26.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.164+0000 7f25bed9d640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25a8004d40 0x7f25a8005150 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f25c00b91c0 tx=0x7f25a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8c73d81de2bf1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:26.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.164+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b000a4c0 con 0x7f25a8004d40 2024-04-25T19:13:26.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.164+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b002f070 con 0x7f25a8004d40 2024-04-25T19:13:26.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.164+0000 7f25bd59a640 1 -- 172.21.15.28:0/3184545670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b002bcd0 con 0x7f25a8004d40 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.164+0000 7f25c4cf0640 1 -- 172.21.15.28:0/3184545670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25a8004d40 msgr2=0x7f25a8005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.164+0000 7f25c4cf0640 1 --2- 172.21.15.28:0/3184545670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25a8004d40 0x7f25a8005150 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f25c00b91c0 tx=0x7f25a8005830 comp rx=0 tx=0).stop 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.164+0000 7f25bed9d640 1 -- 172.21.15.28:0/3184545670 reap_dead start 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.165+0000 7f25c4cf0640 1 -- 172.21.15.28:0/3184545670 shutdown_connections 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.165+0000 7f25c4cf0640 1 -- 172.21.15.28:0/3184545670 >> 172.21.15.28:0/3184545670 conn(0x7f25c00d03a0 msgr2=0x7f25c00d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.165+0000 7f25c4cf0640 1 -- 172.21.15.28:0/3184545670 shutdown_connections 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.165+0000 7f25c4cf0640 1 -- 172.21.15.28:0/3184545670 wait complete. 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.167+0000 7f25c4cf0640 1 Processor -- start 2024-04-25T19:13:26.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.167+0000 7f25c4cf0640 1 -- start start 2024-04-25T19:13:26.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25c4cf0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8004d40 0x7f25c00a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:26.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25c4cf0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c0158d10 0x7f25c00a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:26.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25c4cf0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c0159120 0x7f25c00a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:26.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25c4cf0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f25c00d1040 con 0x7f25c0158d10 2024-04-25T19:13:26.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25c4cf0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f25c00d0ec0 con 0x7f25a8004d40 2024-04-25T19:13:26.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25c4cf0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f25c00b8ce0 con 0x7f25c0159120 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c0158d10 0x7f25c00a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c0158d10 0x7f25c00a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35898/0 (socket says 172.21.15.28:35898) 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 -- 172.21.15.28:0/1580356212 learned_addr learned my addr 172.21.15.28:0/1580356212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 -- 172.21.15.28:0/1580356212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c0159120 msgr2=0x7f25c00a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c0159120 0x7f25c00a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 -- 172.21.15.28:0/1580356212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8004d40 msgr2=0x7f25c00a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25be59c640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8004d40 0x7f25c00a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8004d40 0x7f25c00a0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 -- 172.21.15.28:0/1580356212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c009ddb0 con 0x7f25c0158d10 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.168+0000 7f25bdd9b640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c0158d10 0x7f25c00a0c40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f25b4011950 tx=0x7f25b4011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.169+0000 7f25aeffd640 1 -- 172.21.15.28:0/1580356212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b400faf0 con 0x7f25c0158d10 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.169+0000 7f25aeffd640 1 -- 172.21.15.28:0/1580356212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b400fc90 con 0x7f25c0158d10 2024-04-25T19:13:26.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.169+0000 7f25aeffd640 1 -- 172.21.15.28:0/1580356212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4014440 con 0x7f25c0158d10 2024-04-25T19:13:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.169+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c00cbac0 con 0x7f25c0158d10 2024-04-25T19:13:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.169+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c00cc080 con 0x7f25c0158d10 2024-04-25T19:13:26.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.170+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c0069310 con 0x7f25c0158d10 2024-04-25T19:13:26.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.171+0000 7f25aeffd640 1 -- 172.21.15.28:0/1580356212 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f25b4012040 con 0x7f25c0158d10 2024-04-25T19:13:26.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.171+0000 7f25aeffd640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f25a00744e0 0x7f25a00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:26.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.171+0000 7f25aeffd640 1 -- 172.21.15.28:0/1580356212 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 6669+0+0 (secure 0 0 0) 0x7f25b4095cd0 con 0x7f25c0158d10 2024-04-25T19:13:26.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.171+0000 7f25be59c640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f25a00744e0 0x7f25a00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:26.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.172+0000 7f25be59c640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f25a00744e0 0x7f25a00769f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25b8005670 tx=0x7f25b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:26.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.176+0000 7f25aeffd640 1 -- 172.21.15.28:0/1580356212 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f25c0069310 con 0x7f25c0158d10 2024-04-25T19:13:26.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:26.361+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f25c006a100 con 0x7f25c0158d10 2024-04-25T19:13:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.031+0000 7f25aeffd640 1 -- 172.21.15.28:0/1580356212 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v49) ==== 181+0+0 (secure 0 0 0) 0x7f25b4019030 con 0x7f25c0158d10 2024-04-25T19:13:27.033 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-04-25T19:13:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f25a00744e0 msgr2=0x7f25a00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f25a00744e0 0x7f25a00769f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f25b8005670 tx=0x7f25b8009040 comp rx=0 tx=0).stop 2024-04-25T19:13:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c0158d10 msgr2=0x7f25c00a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c0158d10 0x7f25c00a0c40 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f25b4011950 tx=0x7f25b4011e10 comp rx=0 tx=0).stop 2024-04-25T19:13:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 shutdown_connections 2024-04-25T19:13:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f25a00744e0 0x7f25a00769f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f25c0159120 0x7f25c00a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25c0158d10 0x7f25c00a0c40 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 --2- 172.21.15.28:0/1580356212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25a8004d40 0x7f25c00a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.035+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 >> 172.21.15.28:0/1580356212 conn(0x7f25c00d03a0 msgr2=0x7f25c0072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:27.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.036+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 shutdown_connections 2024-04-25T19:13:27.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.036+0000 7f25c4cf0640 1 -- 172.21.15.28:0/1580356212 wait complete. 2024-04-25T19:13:27.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T19:13:27.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.173+0000 7f8424ab6640 1 Processor -- start 2024-04-25T19:13:27.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.173+0000 7f8424ab6640 1 -- start start 2024-04-25T19:13:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f8424ab6640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84200d5690 0x7f842006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f8424ab6640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84200d12e0 con 0x7f84200b4d00 2024-04-25T19:13:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f8424ab6640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84200d15c0 con 0x7f84200b47d0 2024-04-25T19:13:27.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f8424ab6640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f84200d4bb0 con 0x7f84200d5690 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841ed9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f84200b4d00 0x7f84200b9200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39868/0 (socket says 172.21.15.28:39868) 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841ed9d640 1 -- 172.21.15.28:0/2616726333 learned_addr learned my addr 172.21.15.28:0/2616726333 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841dd9b640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84200d5690 0x7f842006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109013217 0 0) 0x7f84200d12e0 con 0x7f84200b4d00 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8408003520 con 0x7f84200b4d00 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1329591904 0 0) 0x7f8408003520 con 0x7f84200b4d00 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84200d5690 msgr2=0x7f842006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84200d5690 0x7f842006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 >> v1:172.21.15.53:6789/0 conn(0x7f84200b47d0 legacy=0x7f84201601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84200d4e90 con 0x7f84200b4d00 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f8418001a10 con 0x7f84200b4d00 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 >> v1:172.21.15.28:6789/0 conn(0x7f84200b4d00 legacy=0x7f84200b9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84080039e0 0x7f8408003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84200d5690 0x7f8408004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8408004d40 0x7f8408005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84200d4bb0 con 0x7f84080039e0 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f840800bb50 con 0x7f8408004d40 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.175+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f840800bcd0 con 0x7f84200d5690 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841e59c640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84080039e0 0x7f8408003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841dd9b640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84200d5690 0x7f8408004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841ed9d640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8408004d40 0x7f8408005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841e59c640 1 -- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84200d5690 msgr2=0x7f8408004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841e59c640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84200d5690 0x7f8408004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841e59c640 1 -- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8408004d40 msgr2=0x7f8408005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841e59c640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8408004d40 0x7f8408005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841e59c640 1 -- 172.21.15.28:0/2616726333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8408003520 con 0x7f84080039e0 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.176+0000 7f841ed9d640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8408004d40 0x7f8408005170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.177+0000 7f841e59c640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84080039e0 0x7f8408003e50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8410001720 tx=0x7f8408005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a946be7b21eebb61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.177+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8410001370 con 0x7f84080039e0 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.177+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84100052d0 con 0x7f84080039e0 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.177+0000 7f841d59a640 1 -- 172.21.15.28:0/2616726333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841002abc0 con 0x7f84080039e0 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.177+0000 7f8424ab6640 1 -- 172.21.15.28:0/2616726333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84080039e0 msgr2=0x7f8408003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.177+0000 7f8424ab6640 1 --2- 172.21.15.28:0/2616726333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84080039e0 0x7f8408003e50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8410001720 tx=0x7f8408005830 comp rx=0 tx=0).stop 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.177+0000 7f841ed9d640 1 -- 172.21.15.28:0/2616726333 reap_dead start 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.178+0000 7f8424ab6640 1 -- 172.21.15.28:0/2616726333 shutdown_connections 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.178+0000 7f8424ab6640 1 -- 172.21.15.28:0/2616726333 >> 172.21.15.28:0/2616726333 conn(0x7f84200d2c60 msgr2=0x7f84200d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.178+0000 7f8424ab6640 1 -- 172.21.15.28:0/2616726333 shutdown_connections 2024-04-25T19:13:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.178+0000 7f8424ab6640 1 -- 172.21.15.28:0/2616726333 wait complete. 2024-04-25T19:13:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.179+0000 7f8424ab6640 1 Processor -- start 2024-04-25T19:13:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.180+0000 7f8424ab6640 1 -- start start 2024-04-25T19:13:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f8424ab6640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84080039e0 0x7f842014ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f8424ab6640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84201e5670 0x7f84201e5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f8424ab6640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f842015cd70 0x7f842015d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f8424ab6640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f84200b9a90 con 0x7f842015cd70 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f8424ab6640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f84200b9910 con 0x7f84080039e0 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f8424ab6640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f84200b9c10 con 0x7f84201e5670 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f842015cd70 0x7f842015d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f842015cd70 0x7f842015d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35922/0 (socket says 172.21.15.28:35922) 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841ed9d640 1 -- 172.21.15.28:0/1451434469 learned_addr learned my addr 172.21.15.28:0/1451434469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841e59c640 1 --2- 172.21.15.28:0/1451434469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84080039e0 0x7f842014ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841dd9b640 1 --2- 172.21.15.28:0/1451434469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84201e5670 0x7f84201e5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841ed9d640 1 -- 172.21.15.28:0/1451434469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84201e5670 msgr2=0x7f84201e5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841ed9d640 1 --2- 172.21.15.28:0/1451434469 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f84201e5670 0x7f84201e5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841ed9d640 1 -- 172.21.15.28:0/1451434469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84080039e0 msgr2=0x7f842014ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841ed9d640 1 --2- 172.21.15.28:0/1451434469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84080039e0 0x7f842014ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.181+0000 7f841ed9d640 1 -- 172.21.15.28:0/1451434469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f842015d6c0 con 0x7f842015cd70 2024-04-25T19:13:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.182+0000 7f841ed9d640 1 --2- 172.21.15.28:0/1451434469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f842015cd70 0x7f842015d180 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f84200d01b0 tx=0x7f841800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.182+0000 7f840effd640 1 -- 172.21.15.28:0/1451434469 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f841800ec10 con 0x7f842015cd70 2024-04-25T19:13:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.182+0000 7f840effd640 1 -- 172.21.15.28:0/1451434469 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8418018070 con 0x7f842015cd70 2024-04-25T19:13:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.182+0000 7f8424ab6640 1 -- 172.21.15.28:0/1451434469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f842014f370 con 0x7f842015cd70 2024-04-25T19:13:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.182+0000 7f840effd640 1 -- 172.21.15.28:0/1451434469 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8418013420 con 0x7f842015cd70 2024-04-25T19:13:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.182+0000 7f8424ab6640 1 -- 172.21.15.28:0/1451434469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84201e6660 con 0x7f842015cd70 2024-04-25T19:13:27.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.183+0000 7f8424ab6640 1 -- 172.21.15.28:0/1451434469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f842006a100 con 0x7f842015cd70 2024-04-25T19:13:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.183+0000 7f840effd640 1 -- 172.21.15.28:0/1451434469 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f84180135c0 con 0x7f842015cd70 2024-04-25T19:13:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.184+0000 7f840effd640 1 --2- 172.21.15.28:0/1451434469 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f84000744b0 0x7f84000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.184+0000 7f841e59c640 1 --2- 172.21.15.28:0/1451434469 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f84000744b0 0x7f84000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.184+0000 7f840effd640 1 -- 172.21.15.28:0/1451434469 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7f8418095640 con 0x7f842015cd70 2024-04-25T19:13:27.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.185+0000 7f841e59c640 1 --2- 172.21.15.28:0/1451434469 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f84000744b0 0x7f84000769c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8410032000 tx=0x7f841002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.189+0000 7f840effd640 1 -- 172.21.15.28:0/1451434469 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f842015d6c0 con 0x7f842015cd70 2024-04-25T19:13:27.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.374+0000 7f8424ab6640 1 -- 172.21.15.28:0/1451434469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f842015e640 con 0x7f842015cd70 2024-04-25T19:13:27.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.375+0000 7f840effd640 1 -- 172.21.15.28:0/1451434469 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=-16 pool 'cephfs_data' is in use by CephFS v49) ==== 181+0+0 (secure 0 0 0) 0x7f841805fbd0 con 0x7f842015cd70 2024-04-25T19:13:27.377 INFO:teuthology.orchestra.run.smithi028.stderr:Error EBUSY: pool 'cephfs_data' is in use by CephFS 2024-04-25T19:13:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 16 2024-04-25T19:13:27.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:27.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.497+0000 7fbcf97c5640 1 Processor -- start 2024-04-25T19:13:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.497+0000 7fbcf97c5640 1 -- start start 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf97c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 0x7fbcf409b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf97c5640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf415e630 con 0x7fbcf40cfa00 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf97c5640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf409be90 con 0x7fbcf415e220 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf97c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf40cffe0 con 0x7fbcf409b540 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf2ffd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbcf40cfa00 0x7fbcf40b21d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39884/0 (socket says 172.21.15.28:39884) 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf2ffd640 1 -- 172.21.15.28:0/1089642963 learned_addr learned my addr 172.21.15.28:0/1089642963 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf37fe640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 0x7fbcf409b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1139452199 0 0) 0x7fbcf415e630 con 0x7fbcf40cfa00 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.498+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbce0003520 con 0x7fbcf40cfa00 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3897643062 0 0) 0x7fbce0003520 con 0x7fbcf40cfa00 2024-04-25T19:13:27.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 msgr2=0x7fbcf409b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 0x7fbcf409b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 >> v1:172.21.15.53:6789/0 conn(0x7fbcf415e220 legacy=0x7fbcf4157570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf40d02c0 con 0x7fbcf40cfa00 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fbcec001670 con 0x7fbcf40cfa00 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 >> v1:172.21.15.28:6789/0 conn(0x7fbcf40cfa00 legacy=0x7fbcf40b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbce00039e0 0x7fbce0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 0x7fbce0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce0004d40 0x7fbce0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf40cffe0 con 0x7fbce00039e0 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbce000bb80 con 0x7fbce0004d40 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbce000bd80 con 0x7fbcf409b540 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf2ffd640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbce00039e0 0x7fbce0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf37fe640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce0004d40 0x7fbce0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf27fc640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 0x7fbce0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf2ffd640 1 -- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 msgr2=0x7fbce0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf2ffd640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 0x7fbce0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf2ffd640 1 -- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce0004d40 msgr2=0x7fbce0005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf2ffd640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce0004d40 0x7fbce0005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf2ffd640 1 -- 172.21.15.28:0/1089642963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce0003520 con 0x7fbce00039e0 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf37fe640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce0004d40 0x7fbce0005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.499+0000 7fbcf27fc640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf409b540 0x7fbce0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.500+0000 7fbcf2ffd640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbce00039e0 0x7fbce0003e50 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00c210 tx=0x7fbce0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb14ace228a1d03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.500+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec007d10 con 0x7fbce00039e0 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.500+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcec0058d0 con 0x7fbce00039e0 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.500+0000 7fbcf1ffb640 1 -- 172.21.15.28:0/1089642963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcec005d00 con 0x7fbce00039e0 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.501+0000 7fbcf97c5640 1 -- 172.21.15.28:0/1089642963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbce00039e0 msgr2=0x7fbce0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.501+0000 7fbcf97c5640 1 --2- 172.21.15.28:0/1089642963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbce00039e0 0x7fbce0003e50 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fbcec00c210 tx=0x7fbce0005830 comp rx=0 tx=0).stop 2024-04-25T19:13:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.501+0000 7fbcf37fe640 1 -- 172.21.15.28:0/1089642963 reap_dead start 2024-04-25T19:13:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.501+0000 7fbcf97c5640 1 -- 172.21.15.28:0/1089642963 shutdown_connections 2024-04-25T19:13:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.501+0000 7fbcf97c5640 1 -- 172.21.15.28:0/1089642963 >> 172.21.15.28:0/1089642963 conn(0x7fbcf40b7620 msgr2=0x7fbcf40b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.501+0000 7fbcf97c5640 1 -- 172.21.15.28:0/1089642963 shutdown_connections 2024-04-25T19:13:27.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.501+0000 7fbcf97c5640 1 -- 172.21.15.28:0/1089642963 wait complete. 2024-04-25T19:13:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.503+0000 7fbcf97c5640 1 Processor -- start 2024-04-25T19:13:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.503+0000 7fbcf97c5640 1 -- start start 2024-04-25T19:13:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf97c5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce00039e0 0x7fbcf41f3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf97c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf40d17d0 0x7fbcf40d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf97c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf40d2120 0x7fbcf4153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf97c5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf415d800 con 0x7fbcf40d2120 2024-04-25T19:13:27.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf97c5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbcf415d680 con 0x7fbce00039e0 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf97c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbcf409b950 con 0x7fbcf40d17d0 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf40d2120 0x7fbcf4153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf37fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf40d2120 0x7fbcf4153280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35926/0 (socket says 172.21.15.28:35926) 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.504+0000 7fbcf37fe640 1 -- 172.21.15.28:0/3487416875 learned_addr learned my addr 172.21.15.28:0/3487416875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf2ffd640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce00039e0 0x7fbcf41f3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf27fc640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf40d17d0 0x7fbcf40d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf37fe640 1 -- 172.21.15.28:0/3487416875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf40d17d0 msgr2=0x7fbcf40d1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf37fe640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf40d17d0 0x7fbcf40d1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf37fe640 1 -- 172.21.15.28:0/3487416875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce00039e0 msgr2=0x7fbcf41f3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf37fe640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce00039e0 0x7fbcf41f3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf37fe640 1 -- 172.21.15.28:0/3487416875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf4152530 con 0x7fbcf40d2120 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf37fe640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf40d2120 0x7fbcf4153280 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4074100 tx=0x7fbce4010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbceb7fe640 1 -- 172.21.15.28:0/3487416875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4010d80 con 0x7fbcf40d2120 2024-04-25T19:13:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbceb7fe640 1 -- 172.21.15.28:0/3487416875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbce4003990 con 0x7fbcf40d2120 2024-04-25T19:13:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbceb7fe640 1 -- 172.21.15.28:0/3487416875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbce4003dc0 con 0x7fbcf40d2120 2024-04-25T19:13:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf41527a0 con 0x7fbcf40d2120 2024-04-25T19:13:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.505+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcf4152d60 con 0x7fbcf40d2120 2024-04-25T19:13:27.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.506+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcf40b3810 con 0x7fbcf40d2120 2024-04-25T19:13:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.507+0000 7fbceb7fe640 1 -- 172.21.15.28:0/3487416875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbce4005d10 con 0x7fbcf40d2120 2024-04-25T19:13:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.507+0000 7fbceb7fe640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbcd40744b0 0x7fbcd40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.507+0000 7fbceb7fe640 1 -- 172.21.15.28:0/3487416875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7fbce4095ae0 con 0x7fbcf40d2120 2024-04-25T19:13:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.507+0000 7fbcf2ffd640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbcd40744b0 0x7fbcd40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.508+0000 7fbcf2ffd640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbcd40744b0 0x7fbcd40769c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbcec033000 tx=0x7fbcec02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.512+0000 7fbceb7fe640 1 -- 172.21.15.28:0/3487416875 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbcf40b3810 con 0x7fbcf40d2120 2024-04-25T19:13:27.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.698+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbcf4050740 con 0x7fbcf40d2120 2024-04-25T19:13:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.699+0000 7fbceb7fe640 1 -- 172.21.15.28:0/3487416875 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+17017 (secure 0 0 0) 0x7fbce4025020 con 0x7fbcf40d2120 2024-04-25T19:13:27.701 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:27.701 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":49,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:27.025837+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbcd40744b0 msgr2=0x7fbcd40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbcd40744b0 0x7fbcd40769c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbcec033000 tx=0x7fbcec02e000 comp rx=0 tx=0).stop 2024-04-25T19:13:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf40d2120 msgr2=0x7fbcf4153280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf40d2120 0x7fbcf4153280 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4074100 tx=0x7fbce4010590 comp rx=0 tx=0).stop 2024-04-25T19:13:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 shutdown_connections 2024-04-25T19:13:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbcd40744b0 0x7fbcd40769c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf40d2120 0x7fbcf4153280 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbcf40d17d0 0x7fbcf40d1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 --2- 172.21.15.28:0/3487416875 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbce00039e0 0x7fbcf41f3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 >> 172.21.15.28:0/3487416875 conn(0x7fbcf40b7620 msgr2=0x7fbcf40b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 shutdown_connections 2024-04-25T19:13:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.701+0000 7fbcf97c5640 1 -- 172.21.15.28:0/3487416875 wait complete. 2024-04-25T19:13:27.714 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T19:11:16.852809+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T19:12:43.014197+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'recovery_fs_metadata', 'create_time': '2024-04-25T19:13:02.679639+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'recovery_fs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T19:13:27.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T19:13:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.828+0000 7f71a5582640 1 Processor -- start 2024-04-25T19:13:27.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.828+0000 7f71a5582640 1 -- start start 2024-04-25T19:13:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f71a5582640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71a00b4bb0 0x7f71a00ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f71a5582640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a00b2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f71a5582640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71a015f680 con 0x7f71a015f270 2024-04-25T19:13:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f71a5582640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f71a015ede0 con 0x7f71a00b4bb0 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f71a5582640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f71a015e540 con 0x7f71a00cead0 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719e7fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71a00b4bb0 0x7f71a00ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719effd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f71a015f270 0x7f71a00b44a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39892/0 (socket says 172.21.15.28:39892) 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719e7fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71a00b4bb0 0x7f71a00ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35938/0 (socket says 172.21.15.28:35938) 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719effd640 1 -- 172.21.15.28:0/2530331424 learned_addr learned my addr 172.21.15.28:0/2530331424 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719f7fe640 1 --2- 172.21.15.28:0/2530331424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a00b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719e7fc640 1 -- 172.21.15.28:0/2530331424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 msgr2=0x7f71a00b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719e7fc640 1 --2- 172.21.15.28:0/2530331424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a00b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719dffb640 1 -- 172.21.15.28:0/2530331424 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1980189476 0 0) 0x7f71a015f680 con 0x7f71a015f270 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719e7fc640 1 -- 172.21.15.28:0/2530331424 >> v1:172.21.15.28:6789/0 conn(0x7f71a015f270 legacy=0x7f71a00b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719e7fc640 1 -- 172.21.15.28:0/2530331424 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a00cf2e0 con 0x7f71a00b4bb0 2024-04-25T19:13:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.829+0000 7f719f7fe640 1 --2- 172.21.15.28:0/2530331424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a00b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.830+0000 7f719e7fc640 1 --2- 172.21.15.28:0/2530331424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00b4bb0 0x7f71a00ce590 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f71880097a0 tx=0x7f718802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a7588feeb518c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.830+0000 7f719dffb640 1 -- 172.21.15.28:0/2530331424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188037070 con 0x7f71a00b4bb0 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.830+0000 7f719dffb640 1 -- 172.21.15.28:0/2530331424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7188032440 con 0x7f71a00b4bb0 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.830+0000 7f719dffb640 1 -- 172.21.15.28:0/2530331424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7188032810 con 0x7f71a00b4bb0 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.831+0000 7f71a5582640 1 -- 172.21.15.28:0/2530331424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00b4bb0 msgr2=0x7f71a00ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.831+0000 7f71a5582640 1 --2- 172.21.15.28:0/2530331424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00b4bb0 0x7f71a00ce590 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f71880097a0 tx=0x7f718802a180 comp rx=0 tx=0).stop 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.831+0000 7f71a5582640 1 -- 172.21.15.28:0/2530331424 shutdown_connections 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.831+0000 7f71a5582640 1 --2- 172.21.15.28:0/2530331424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a00b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.831+0000 7f71a5582640 1 --2- 172.21.15.28:0/2530331424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00b4bb0 0x7f71a00ce590 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.831+0000 7f71a5582640 1 -- 172.21.15.28:0/2530331424 >> 172.21.15.28:0/2530331424 conn(0x7f71a00b7640 msgr2=0x7f71a00b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.831+0000 7f71a5582640 1 -- 172.21.15.28:0/2530331424 shutdown_connections 2024-04-25T19:13:27.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.831+0000 7f71a5582640 1 -- 172.21.15.28:0/2530331424 wait complete. 2024-04-25T19:13:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.833+0000 7f71a5582640 1 Processor -- start 2024-04-25T19:13:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.833+0000 7f71a5582640 1 -- start start 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f71a5582640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a0155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f71a5582640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a015f270 0x7f71a00c8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f71a5582640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00c6bf0 0x7f71a00c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f71a5582640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71a00ceee0 con 0x7f71a00c6bf0 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f71a5582640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f71a015ea70 con 0x7f71a015f270 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f71a5582640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f71a00cf060 con 0x7f71a00cead0 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f719f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00c6bf0 0x7f71a00c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f719f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00c6bf0 0x7f71a00c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35944/0 (socket says 172.21.15.28:35944) 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.834+0000 7f719f7fe640 1 -- 172.21.15.28:0/2981753388 learned_addr learned my addr 172.21.15.28:0/2981753388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:27.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719effd640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a0155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719e7fc640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a015f270 0x7f71a00c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719f7fe640 1 -- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 msgr2=0x7f71a0155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719f7fe640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a0155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719f7fe640 1 -- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a015f270 msgr2=0x7f71a00c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719f7fe640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a015f270 0x7f71a00c8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719f7fe640 1 -- 172.21.15.28:0/2981753388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71a0155bd0 con 0x7f71a00c6bf0 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719effd640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a0155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719e7fc640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a015f270 0x7f71a00c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f719f7fe640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00c6bf0 0x7f71a00c8c70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f719000cd40 tx=0x7f719000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f71977fe640 1 -- 172.21.15.28:0/2981753388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7190003b90 con 0x7f71a00c6bf0 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f71977fe640 1 -- 172.21.15.28:0/2981753388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7190003d50 con 0x7f71a00c6bf0 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71a00c9980 con 0x7f71a00c6bf0 2024-04-25T19:13:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f71977fe640 1 -- 172.21.15.28:0/2981753388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7190004260 con 0x7f71a00c6bf0 2024-04-25T19:13:27.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.835+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71a01562c0 con 0x7f71a00c6bf0 2024-04-25T19:13:27.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.836+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71a0156030 con 0x7f71a00c6bf0 2024-04-25T19:13:27.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.837+0000 7f71977fe640 1 -- 172.21.15.28:0/2981753388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7190004400 con 0x7f71a00c6bf0 2024-04-25T19:13:27.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.837+0000 7f71977fe640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71800744b0 0x7f71800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:27.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.838+0000 7f719effd640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71800744b0 0x7f71800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:27.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.838+0000 7f71977fe640 1 -- 172.21.15.28:0/2981753388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7f7190095e30 con 0x7f71a00c6bf0 2024-04-25T19:13:27.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.838+0000 7f719effd640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71800744b0 0x7f71800769c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7198000f60 tx=0x7f7198009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:27.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:27.842+0000 7f71977fe640 1 -- 172.21.15.28:0/2981753388 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7190014030 con 0x7f71a00c6bf0 2024-04-25T19:13:28.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.030+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f71a0050740 con 0x7f71a00c6bf0 2024-04-25T19:13:28.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.030+0000 7f71977fe640 1 -- 172.21.15.28:0/2981753388 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v49) ==== 93+0+10 (secure 0 0 0) 0x7f71900603e0 con 0x7f71a00c6bf0 2024-04-25T19:13:28.032 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71800744b0 msgr2=0x7f71800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71800744b0 0x7f71800769c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7198000f60 tx=0x7f7198009040 comp rx=0 tx=0).stop 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00c6bf0 msgr2=0x7f71a00c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00c6bf0 0x7f71a00c8c70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f719000cd40 tx=0x7f719000a650 comp rx=0 tx=0).stop 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 shutdown_connections 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71800744b0 0x7f71800769c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71a00c6bf0 0x7f71a00c8c70 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71a015f270 0x7f71a00c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 --2- 172.21.15.28:0/2981753388 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71a00cead0 0x7f71a0155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 >> 172.21.15.28:0/2981753388 conn(0x7f71a00b7640 msgr2=0x7f71a00b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 shutdown_connections 2024-04-25T19:13:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.033+0000 7f71a5582640 1 -- 172.21.15.28:0/2981753388 wait complete. 2024-04-25T19:13:28.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T19:13:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.159+0000 7f78ffd84640 1 Processor -- start 2024-04-25T19:13:28.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.159+0000 7f78ffd84640 1 -- start start 2024-04-25T19:13:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78ffd84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f8160310 0x7f78f80d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78ffd84640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78f80b79f0 0x7f78f80b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78ffd84640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f8160720 con 0x7f78f808cce0 2024-04-25T19:13:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78ffd84640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f78f80b8340 con 0x7f78f80b79f0 2024-04-25T19:13:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78ffd84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f78f80d5640 con 0x7f78f8160310 2024-04-25T19:13:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78fd31f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78f80b79f0 0x7f78f80b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78fd31f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78f80b79f0 0x7f78f80b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35952/0 (socket says 172.21.15.28:35952) 2024-04-25T19:13:28.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78fd31f640 1 -- 172.21.15.28:0/199105900 learned_addr learned my addr 172.21.15.28:0/199105900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78fdb20640 1 --2- 172.21.15.28:0/199105900 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f8160310 0x7f78f80d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78fd31f640 1 -- 172.21.15.28:0/199105900 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f8160310 msgr2=0x7f78f80d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78fd31f640 1 --2- 172.21.15.28:0/199105900 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f8160310 0x7f78f80d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78fd31f640 1 -- 172.21.15.28:0/199105900 >> v1:172.21.15.53:6789/0 conn(0x7f78f808cce0 legacy=0x7f78f808d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.161+0000 7f78fd31f640 1 -- 172.21.15.28:0/199105900 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f80d0f40 con 0x7f78f80b79f0 2024-04-25T19:13:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.162+0000 7f78fd31f640 1 --2- 172.21.15.28:0/199105900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f80b79f0 0x7f78f80b7e00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009870 tx=0x7f78e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c852dfde509aaec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.162+0000 7f78fcb1e640 1 -- 172.21.15.28:0/199105900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8037070 con 0x7f78f80b79f0 2024-04-25T19:13:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.162+0000 7f78fcb1e640 1 -- 172.21.15.28:0/199105900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e8032440 con 0x7f78f80b79f0 2024-04-25T19:13:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.162+0000 7f78fcb1e640 1 -- 172.21.15.28:0/199105900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8032810 con 0x7f78f80b79f0 2024-04-25T19:13:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.163+0000 7f78ffd84640 1 -- 172.21.15.28:0/199105900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f80b79f0 msgr2=0x7f78f80b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.163+0000 7f78ffd84640 1 --2- 172.21.15.28:0/199105900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f80b79f0 0x7f78f80b7e00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009870 tx=0x7f78e802a270 comp rx=0 tx=0).stop 2024-04-25T19:13:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.163+0000 7f78ffd84640 1 -- 172.21.15.28:0/199105900 shutdown_connections 2024-04-25T19:13:28.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.163+0000 7f78ffd84640 1 --2- 172.21.15.28:0/199105900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f80b79f0 0x7f78f80b7e00 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.163+0000 7f78ffd84640 1 --2- 172.21.15.28:0/199105900 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f8160310 0x7f78f80d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.163+0000 7f78ffd84640 1 -- 172.21.15.28:0/199105900 >> 172.21.15.28:0/199105900 conn(0x7f78f80d2a50 msgr2=0x7f78f80d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.163+0000 7f78ffd84640 1 -- 172.21.15.28:0/199105900 shutdown_connections 2024-04-25T19:13:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.163+0000 7f78ffd84640 1 -- 172.21.15.28:0/199105900 wait complete. 2024-04-25T19:13:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.165+0000 7f78ffd84640 1 Processor -- start 2024-04-25T19:13:28.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.165+0000 7f78ffd84640 1 -- start start 2024-04-25T19:13:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78ffd84640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f808cce0 0x7f78f80cee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78ffd84640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f80b79f0 0x7f78f8155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78ffd84640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f8160310 0x7f78f80cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78ffd84640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78f80b9400 con 0x7f78f8160310 2024-04-25T19:13:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78ffd84640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f78f80d1500 con 0x7f78f80b79f0 2024-04-25T19:13:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78ffd84640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f78f80d0bf0 con 0x7f78f808cce0 2024-04-25T19:13:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fe321640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f8160310 0x7f78f80cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fe321640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f8160310 0x7f78f80cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35954/0 (socket says 172.21.15.28:35954) 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fe321640 1 -- 172.21.15.28:0/4171283966 learned_addr learned my addr 172.21.15.28:0/4171283966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fdb20640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f808cce0 0x7f78f80cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fd31f640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f80b79f0 0x7f78f8155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fe321640 1 -- 172.21.15.28:0/4171283966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f808cce0 msgr2=0x7f78f80cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fe321640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f808cce0 0x7f78f80cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fe321640 1 -- 172.21.15.28:0/4171283966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f80b79f0 msgr2=0x7f78f8155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fe321640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f80b79f0 0x7f78f8155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.166+0000 7f78fe321640 1 -- 172.21.15.28:0/4171283966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f8156260 con 0x7f78f8160310 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.167+0000 7f78fe321640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f8160310 0x7f78f80cdcd0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f78f000c6b0 tx=0x7f78f000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.167+0000 7f78ee7fc640 1 -- 172.21.15.28:0/4171283966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f0003b70 con 0x7f78f8160310 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.167+0000 7f78ee7fc640 1 -- 172.21.15.28:0/4171283966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78f0003d10 con 0x7f78f8160310 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.167+0000 7f78ee7fc640 1 -- 172.21.15.28:0/4171283966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78f0005310 con 0x7f78f8160310 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.167+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f80cf820 con 0x7f78f8160310 2024-04-25T19:13:28.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.167+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f80cb830 con 0x7f78f8160310 2024-04-25T19:13:28.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.168+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f8159650 con 0x7f78f8160310 2024-04-25T19:13:28.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.169+0000 7f78ee7fc640 1 -- 172.21.15.28:0/4171283966 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f78f00054b0 con 0x7f78f8160310 2024-04-25T19:13:28.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.169+0000 7f78ee7fc640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78dc0744b0 0x7f78dc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.169+0000 7f78ee7fc640 1 -- 172.21.15.28:0/4171283966 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7f78f0095120 con 0x7f78f8160310 2024-04-25T19:13:28.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.169+0000 7f78fdb20640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78dc0744b0 0x7f78dc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.170+0000 7f78fdb20640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78dc0744b0 0x7f78dc0769c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009aa0 tx=0x7f78f4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.174+0000 7f78ee7fc640 1 -- 172.21.15.28:0/4171283966 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78f005f050 con 0x7f78f8160310 2024-04-25T19:13:28.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.356+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78f80cfd40 con 0x7f78f8160310 2024-04-25T19:13:28.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.357+0000 7f78ee7fc640 1 -- 172.21.15.28:0/4171283966 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v49) ==== 100+0+11 (secure 0 0 0) 0x7f78f80cfd40 con 0x7f78f8160310 2024-04-25T19:13:28.359 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.358+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78dc0744b0 msgr2=0x7f78dc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.358+0000 7f78ffd84640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78dc0744b0 0x7f78dc0769c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f78f4009aa0 tx=0x7f78f4009190 comp rx=0 tx=0).stop 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f8160310 msgr2=0x7f78f80cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f8160310 0x7f78f80cdcd0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f78f000c6b0 tx=0x7f78f000cb70 comp rx=0 tx=0).stop 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 shutdown_connections 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f78dc0744b0 0x7f78dc0769c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78f8160310 0x7f78f80cdcd0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f80b79f0 0x7f78f8155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 --2- 172.21.15.28:0/4171283966 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f78f808cce0 0x7f78f80cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 >> 172.21.15.28:0/4171283966 conn(0x7f78f80d2a50 msgr2=0x7f78f815f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:28.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 shutdown_connections 2024-04-25T19:13:28.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.359+0000 7f78ffd84640 1 -- 172.21.15.28:0/4171283966 wait complete. 2024-04-25T19:13:28.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get recovery_fs_metadata pg_num 2024-04-25T19:13:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.485+0000 7fdbff534640 1 Processor -- start 2024-04-25T19:13:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.485+0000 7fdbff534640 1 -- start start 2024-04-25T19:13:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.486+0000 7fdbff534640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdbf8160310 0x7fdbf80d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.486+0000 7fdbff534640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbf8160720 con 0x7fdbf80b79f0 2024-04-25T19:13:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.486+0000 7fdbff534640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbf80d5640 con 0x7fdbf80b9920 2024-04-25T19:13:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.486+0000 7fdbff534640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fdbf80d0f40 con 0x7fdbf8160310 2024-04-25T19:13:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.486+0000 7fdbfd2d0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdbf8160310 0x7fdbf80d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.487+0000 7fdbfcacf640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdbf80b79f0 0x7fdbf80b7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39896/0 (socket says 172.21.15.28:39896) 2024-04-25T19:13:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.487+0000 7fdbfd2d0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdbf8160310 0x7fdbf80d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35960/0 (socket says 172.21.15.28:35960) 2024-04-25T19:13:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.487+0000 7fdbfcacf640 1 -- 172.21.15.28:0/318864256 learned_addr learned my addr 172.21.15.28:0/318864256 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.487+0000 7fdbfd2d0640 1 -- 172.21.15.28:0/318864256 >> v1:172.21.15.53:6789/0 conn(0x7fdbf80b9920 legacy=0x7fdbf80b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.487+0000 7fdbeffff640 1 -- 172.21.15.28:0/318864256 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 423395090 0 0) 0x7fdbf8160720 con 0x7fdbf80b79f0 2024-04-25T19:13:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.487+0000 7fdbfd2d0640 1 -- 172.21.15.28:0/318864256 >> v1:172.21.15.28:6789/0 conn(0x7fdbf80b79f0 legacy=0x7fdbf80b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.487+0000 7fdbfd2d0640 1 -- 172.21.15.28:0/318864256 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf80d1220 con 0x7fdbf8160310 2024-04-25T19:13:28.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.488+0000 7fdbfd2d0640 1 --2- 172.21.15.28:0/318864256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80d5100 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40097a0 tx=0x7fdbf402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb03f18493be716a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.488+0000 7fdbeffff640 1 -- 172.21.15.28:0/318864256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4037070 con 0x7fdbf8160310 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.488+0000 7fdbeffff640 1 -- 172.21.15.28:0/318864256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf4032440 con 0x7fdbf8160310 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.488+0000 7fdbeffff640 1 -- 172.21.15.28:0/318864256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4032810 con 0x7fdbf8160310 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.488+0000 7fdbff534640 1 -- 172.21.15.28:0/318864256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 msgr2=0x7fdbf80d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.488+0000 7fdbff534640 1 --2- 172.21.15.28:0/318864256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80d5100 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fdbf40097a0 tx=0x7fdbf402a230 comp rx=0 tx=0).stop 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.489+0000 7fdbff534640 1 -- 172.21.15.28:0/318864256 shutdown_connections 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.489+0000 7fdbff534640 1 --2- 172.21.15.28:0/318864256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80d5100 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.489+0000 7fdbff534640 1 -- 172.21.15.28:0/318864256 >> 172.21.15.28:0/318864256 conn(0x7fdbf80d2a50 msgr2=0x7fdbf80d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.489+0000 7fdbff534640 1 -- 172.21.15.28:0/318864256 shutdown_connections 2024-04-25T19:13:28.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.489+0000 7fdbff534640 1 -- 172.21.15.28:0/318864256 wait complete. 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.490+0000 7fdbff534640 1 Processor -- start 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.490+0000 7fdbff534640 1 -- start start 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbff534640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbf80b79f0 0x7fdbf80cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbff534640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbf80b9920 0x7fdbf80c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbff534640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbff534640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf80d0bf0 con 0x7fdbf8160310 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbff534640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fdbf80b9400 con 0x7fdbf80b79f0 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbff534640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fdbf80cc950 con 0x7fdbf80b9920 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35966/0 (socket says 172.21.15.28:35966) 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 -- 172.21.15.28:0/2390864298 learned_addr learned my addr 172.21.15.28:0/2390864298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfd2d0640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbf80b79f0 0x7fdbf80cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfcacf640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbf80b9920 0x7fdbf80c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 -- 172.21.15.28:0/2390864298 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbf80b9920 msgr2=0x7fdbf80c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbf80b9920 0x7fdbf80c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 -- 172.21.15.28:0/2390864298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbf80b79f0 msgr2=0x7fdbf80cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbf80b79f0 0x7fdbf80cb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 -- 172.21.15.28:0/2390864298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbf80cbef0 con 0x7fdbf8160310 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbfdad1640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80c9c20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000c6e0 tx=0x7fdbf000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbedffb640 1 -- 172.21.15.28:0/2390864298 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0003b70 con 0x7fdbf8160310 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbedffb640 1 -- 172.21.15.28:0/2390864298 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbf0003d30 con 0x7fdbf8160310 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.492+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbf80cc1d0 con 0x7fdbf8160310 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.493+0000 7fdbedffb640 1 -- 172.21.15.28:0/2390864298 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0005310 con 0x7fdbf8160310 2024-04-25T19:13:28.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.493+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbf80cc4b0 con 0x7fdbf8160310 2024-04-25T19:13:28.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.494+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbf80cbc60 con 0x7fdbf8160310 2024-04-25T19:13:28.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.494+0000 7fdbedffb640 1 -- 172.21.15.28:0/2390864298 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdbf00054b0 con 0x7fdbf8160310 2024-04-25T19:13:28.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.495+0000 7fdbedffb640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdbdc0744e0 0x7fdbdc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.495+0000 7fdbfd2d0640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdbdc0744e0 0x7fdbdc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.495+0000 7fdbedffb640 1 -- 172.21.15.28:0/2390864298 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7fdbf00950c0 con 0x7fdbf8160310 2024-04-25T19:13:28.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.496+0000 7fdbfd2d0640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdbdc0744e0 0x7fdbdc0769f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4034000 tx=0x7fdbf402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.500+0000 7fdbedffb640 1 -- 172.21.15.28:0/2390864298 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdbf005f070 con 0x7fdbf8160310 2024-04-25T19:13:28.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.684+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"} v 0) -- 0x7fdbf8050740 con 0x7fdbf8160310 2024-04-25T19:13:28.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.684+0000 7fdbedffb640 1 -- 172.21.15.28:0/2390864298 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "recovery_fs_metadata", "var": "pg_num"}]=0 v49) ==== 109+0+11 (secure 0 0 0) 0x7fdbf0063370 con 0x7fdbf8160310 2024-04-25T19:13:28.686 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-25T19:13:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdbdc0744e0 msgr2=0x7fdbdc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdbdc0744e0 0x7fdbdc0769f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4034000 tx=0x7fdbf402e000 comp rx=0 tx=0).stop 2024-04-25T19:13:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 msgr2=0x7fdbf80c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80c9c20 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fdbf000c6e0 tx=0x7fdbf000cba0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 shutdown_connections 2024-04-25T19:13:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fdbdc0744e0 0x7fdbdc0769f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbf8160310 0x7fdbf80c9c20 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fdbf80b9920 0x7fdbf80c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 --2- 172.21.15.28:0/2390864298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbf80b79f0 0x7fdbf80cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 >> 172.21.15.28:0/2390864298 conn(0x7fdbf80d2a50 msgr2=0x7fdbf815f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 shutdown_connections 2024-04-25T19:13:28.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.686+0000 7fdbff534640 1 -- 172.21.15.28:0/2390864298 wait complete. 2024-04-25T19:13:28.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:28.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.809+0000 7f53c9a92640 1 Processor -- start 2024-04-25T19:13:28.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.809+0000 7f53c9a92640 1 -- start start 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c9a92640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53c406edb0 0x7f53c40b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c9a92640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 0x7f53c409b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c9a92640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53c40d1b30 0x7f53c40d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c9a92640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f53c409c190 con 0x7f53c40d1b30 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c9a92640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f53c4160c00 con 0x7f53c406edb0 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c9a92640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f53c409beb0 con 0x7f53c41616a0 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c3fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53c40d1b30 0x7f53c40d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c3fff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53c40d1b30 0x7f53c40d1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35974/0 (socket says 172.21.15.28:35974) 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.810+0000 7f53c3fff640 1 -- 172.21.15.28:0/3771444011 learned_addr learned my addr 172.21.15.28:0/3771444011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.811+0000 7f53c2ffd640 1 --2- 172.21.15.28:0/3771444011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 0x7f53c409b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.811+0000 7f53c37fe640 1 --2- 172.21.15.28:0/3771444011 >> v2:172.21.15.53:3300/0 conn(0x7f53c406edb0 0x7f53c40b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.811+0000 7f53c3fff640 1 -- 172.21.15.28:0/3771444011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 msgr2=0x7f53c409b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:28.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.811+0000 7f53c3fff640 1 --2- 172.21.15.28:0/3771444011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 0x7f53c409b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.811+0000 7f53c3fff640 1 -- 172.21.15.28:0/3771444011 >> v2:172.21.15.53:3300/0 conn(0x7f53c406edb0 msgr2=0x7f53c40b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:28.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.811+0000 7f53c3fff640 1 --2- 172.21.15.28:0/3771444011 >> v2:172.21.15.53:3300/0 conn(0x7f53c406edb0 0x7f53c40b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.811+0000 7f53c3fff640 1 -- 172.21.15.28:0/3771444011 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c4160ea0 con 0x7f53c40d1b30 2024-04-25T19:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c3fff640 1 --2- 172.21.15.28:0/3771444011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c40d1f40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f53c40d3980 tx=0x7f53bc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20614bcfa5ebbcfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c27fc640 1 -- 172.21.15.28:0/3771444011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc036070 con 0x7f53c40d1b30 2024-04-25T19:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c27fc640 1 -- 172.21.15.28:0/3771444011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53bc031600 con 0x7f53c40d1b30 2024-04-25T19:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c27fc640 1 -- 172.21.15.28:0/3771444011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53bc0319d0 con 0x7f53c40d1b30 2024-04-25T19:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c9a92640 1 -- 172.21.15.28:0/3771444011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 msgr2=0x7f53c40d1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c9a92640 1 --2- 172.21.15.28:0/3771444011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c40d1f40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f53c40d3980 tx=0x7f53bc02a6a0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c9a92640 1 -- 172.21.15.28:0/3771444011 shutdown_connections 2024-04-25T19:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c9a92640 1 --2- 172.21.15.28:0/3771444011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c40d1f40 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c9a92640 1 --2- 172.21.15.28:0/3771444011 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 0x7f53c409b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c9a92640 1 --2- 172.21.15.28:0/3771444011 >> v2:172.21.15.53:3300/0 conn(0x7f53c406edb0 0x7f53c40b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.812+0000 7f53c9a92640 1 -- 172.21.15.28:0/3771444011 >> 172.21.15.28:0/3771444011 conn(0x7f53c40b7620 msgr2=0x7f53c40b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.813+0000 7f53c9a92640 1 -- 172.21.15.28:0/3771444011 shutdown_connections 2024-04-25T19:13:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.813+0000 7f53c9a92640 1 -- 172.21.15.28:0/3771444011 wait complete. 2024-04-25T19:13:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.814+0000 7f53c9a92640 1 Processor -- start 2024-04-25T19:13:28.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.814+0000 7f53c9a92640 1 -- start start 2024-04-25T19:13:28.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c9a92640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c406edb0 0x7f53c4161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c9a92640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c414f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c9a92640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 0x7f53c41579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c9a92640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f53c40d2480 con 0x7f53c40d1b30 2024-04-25T19:13:28.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c9a92640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f53c40b4180 con 0x7f53c406edb0 2024-04-25T19:13:28.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c9a92640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f53c409ba80 con 0x7f53c41616a0 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c414f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c414f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35980/0 (socket says 172.21.15.28:35980) 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 -- 172.21.15.28:0/10564703 learned_addr learned my addr 172.21.15.28:0/10564703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c37fe640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c406edb0 0x7f53c4161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c3fff640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 0x7f53c41579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 -- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 msgr2=0x7f53c41579b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 0x7f53c41579b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 -- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c406edb0 msgr2=0x7f53c4161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:28.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c406edb0 0x7f53c4161ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 -- 172.21.15.28:0/10564703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53c4162300 con 0x7f53c40d1b30 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c37fe640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c406edb0 0x7f53c4161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53c2ffd640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c414f2d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f53b800e870 tx=0x7f53b800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.816+0000 7f53abfff640 1 -- 172.21.15.28:0/10564703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b8017070 con 0x7f53c40d1b30 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.817+0000 7f53abfff640 1 -- 172.21.15.28:0/10564703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53b800ee60 con 0x7f53c40d1b30 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.817+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53c41625e0 con 0x7f53c40d1b30 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.817+0000 7f53abfff640 1 -- 172.21.15.28:0/10564703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b800cb60 con 0x7f53c40d1b30 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.817+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53c4162ba0 con 0x7f53c40d1b30 2024-04-25T19:13:28.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.817+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53c414f810 con 0x7f53c40d1b30 2024-04-25T19:13:28.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.818+0000 7f53abfff640 1 -- 172.21.15.28:0/10564703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f53b801a070 con 0x7f53c40d1b30 2024-04-25T19:13:28.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.819+0000 7f53abfff640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53b00744b0 0x7f53b00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:28.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.819+0000 7f53abfff640 1 -- 172.21.15.28:0/10564703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7f53b8095590 con 0x7f53c40d1b30 2024-04-25T19:13:28.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.819+0000 7f53c37fe640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53b00744b0 0x7f53b00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:28.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.819+0000 7f53c37fe640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53b00744b0 0x7f53b00769c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f53c406e970 tx=0x7f53b4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:28.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:28.823+0000 7f53abfff640 1 -- 172.21.15.28:0/10564703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53c414f810 con 0x7f53c40d1b30 2024-04-25T19:13:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.050+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53c4050740 con 0x7f53c40d1b30 2024-04-25T19:13:29.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.050+0000 7f53abfff640 1 -- 172.21.15.28:0/10564703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5184 (secure 0 0 0) 0x7f53b805fc30 con 0x7f53c40d1b30 2024-04-25T19:13:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:29.053 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:29.053 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53b00744b0 msgr2=0x7f53b00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53b00744b0 0x7f53b00769c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f53c406e970 tx=0x7f53b4008040 comp rx=0 tx=0).stop 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 msgr2=0x7f53c414f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c414f2d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f53b800e870 tx=0x7f53b800ed30 comp rx=0 tx=0).stop 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 shutdown_connections 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f53b00744b0 0x7f53b00769c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f53c41616a0 0x7f53c41579b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c40d1b30 0x7f53c414f2d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 --2- 172.21.15.28:0/10564703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c406edb0 0x7f53c4161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 >> 172.21.15.28:0/10564703 conn(0x7f53c40b7620 msgr2=0x7f53c4163f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 shutdown_connections 2024-04-25T19:13:29.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.053+0000 7f53c9a92640 1 -- 172.21.15.28:0/10564703 wait complete. 2024-04-25T19:13:29.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.175+0000 7f38b0f38640 1 Processor -- start 2024-04-25T19:13:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.175+0000 7f38b0f38640 1 -- start start 2024-04-25T19:13:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38b0f38640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac06d580 0x7f38ac06d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38b0f38640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38ac06f1e0 0x7f38ac06f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38b0f38640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38ac06ded0 con 0x7f38ac15f180 2024-04-25T19:13:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38b0f38640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f38ac15f590 con 0x7f38ac06f1e0 2024-04-25T19:13:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38b0f38640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac158ae0 con 0x7f38ac06d580 2024-04-25T19:13:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38aad9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38ac06f1e0 0x7f38ac06f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38aad9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38ac06f1e0 0x7f38ac06f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35988/0 (socket says 172.21.15.28:35988) 2024-04-25T19:13:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38aad9d640 1 -- 172.21.15.28:0/2050882015 learned_addr learned my addr 172.21.15.28:0/2050882015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38aad9d640 1 -- 172.21.15.28:0/2050882015 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac06d580 msgr2=0x7f38ac06d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38aad9d640 1 --2- 172.21.15.28:0/2050882015 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac06d580 0x7f38ac06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38aad9d640 1 -- 172.21.15.28:0/2050882015 >> v1:172.21.15.53:6789/0 conn(0x7f38ac15f180 legacy=0x7f38ac161c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:29.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.177+0000 7f38aad9d640 1 -- 172.21.15.28:0/2050882015 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac162700 con 0x7f38ac06f1e0 2024-04-25T19:13:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.178+0000 7f38aad9d640 1 --2- 172.21.15.28:0/2050882015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06f1e0 0x7f38ac06f5f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f389c009e80 tx=0x7f389c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba44c8190a5c688 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.178+0000 7f38a959a640 1 -- 172.21.15.28:0/2050882015 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c036070 con 0x7f38ac06f1e0 2024-04-25T19:13:29.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.178+0000 7f38a959a640 1 -- 172.21.15.28:0/2050882015 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f389c031600 con 0x7f38ac06f1e0 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.178+0000 7f38a959a640 1 -- 172.21.15.28:0/2050882015 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389c0319d0 con 0x7f38ac06f1e0 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.179+0000 7f38b0f38640 1 -- 172.21.15.28:0/2050882015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06f1e0 msgr2=0x7f38ac06f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.179+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2050882015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06f1e0 0x7f38ac06f5f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f389c009e80 tx=0x7f389c02a6f0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.179+0000 7f38b0f38640 1 -- 172.21.15.28:0/2050882015 shutdown_connections 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.179+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2050882015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06f1e0 0x7f38ac06f5f0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.179+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2050882015 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac06d580 0x7f38ac06d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.179+0000 7f38b0f38640 1 -- 172.21.15.28:0/2050882015 >> 172.21.15.28:0/2050882015 conn(0x7f38ac0732b0 msgr2=0x7f38ac0736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.179+0000 7f38b0f38640 1 -- 172.21.15.28:0/2050882015 shutdown_connections 2024-04-25T19:13:29.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.179+0000 7f38b0f38640 1 -- 172.21.15.28:0/2050882015 wait complete. 2024-04-25T19:13:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.181+0000 7f38b0f38640 1 Processor -- start 2024-04-25T19:13:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.181+0000 7f38b0f38640 1 -- start start 2024-04-25T19:13:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.182+0000 7f38b0f38640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06d580 0x7f38ac09b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.182+0000 7f38b0f38640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38ac06f1e0 0x7f38ac1ee470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.182+0000 7f38b0f38640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac15f180 0x7f38ac1607c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.182+0000 7f38b0f38640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac15ec60 con 0x7f38ac06d580 2024-04-25T19:13:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.182+0000 7f38b0f38640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f38ac15e980 con 0x7f38ac06f1e0 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.182+0000 7f38b0f38640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f38ac15e670 con 0x7f38ac15f180 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.182+0000 7f38aa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06d580 0x7f38ac09b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aa59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06d580 0x7f38ac09b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35990/0 (socket says 172.21.15.28:35990) 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aa59c640 1 -- 172.21.15.28:0/2468746042 learned_addr learned my addr 172.21.15.28:0/2468746042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aa59c640 1 -- 172.21.15.28:0/2468746042 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac15f180 msgr2=0x7f38ac1607c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aad9d640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac15f180 0x7f38ac1607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38a9d9b640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38ac06f1e0 0x7f38ac1ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aa59c640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac15f180 0x7f38ac1607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aa59c640 1 -- 172.21.15.28:0/2468746042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38ac06f1e0 msgr2=0x7f38ac1ee470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aa59c640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38ac06f1e0 0x7f38ac1ee470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aa59c640 1 -- 172.21.15.28:0/2468746042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ac1f4690 con 0x7f38ac06d580 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38aa59c640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06d580 0x7f38ac09b540 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f38a400b3c0 tx=0x7f38a400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38a2ffd640 1 -- 172.21.15.28:0/2468746042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4011830 con 0x7f38ac06d580 2024-04-25T19:13:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38a2ffd640 1 -- 172.21.15.28:0/2468746042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a4016070 con 0x7f38ac06d580 2024-04-25T19:13:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38ac1f4970 con 0x7f38ac06d580 2024-04-25T19:13:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38a2ffd640 1 -- 172.21.15.28:0/2468746042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4011c10 con 0x7f38ac06d580 2024-04-25T19:13:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.183+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38ac1f4f30 con 0x7f38ac06d580 2024-04-25T19:13:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.184+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38ac050740 con 0x7f38ac06d580 2024-04-25T19:13:29.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.185+0000 7f38a2ffd640 1 -- 172.21.15.28:0/2468746042 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f38a401b460 con 0x7f38ac06d580 2024-04-25T19:13:29.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.185+0000 7f38a2ffd640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f388c0744b0 0x7f388c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.185+0000 7f38a2ffd640 1 -- 172.21.15.28:0/2468746042 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7f38a4094fc0 con 0x7f38ac06d580 2024-04-25T19:13:29.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.185+0000 7f38a9d9b640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f388c0744b0 0x7f388c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.186+0000 7f38a9d9b640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f388c0744b0 0x7f388c0769c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f38ac1549e0 tx=0x7f3894008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.190+0000 7f38a2ffd640 1 -- 172.21.15.28:0/2468746042 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38a4020020 con 0x7f38ac06d580 2024-04-25T19:13:29.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.378+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38ac074250 con 0x7f38ac06d580 2024-04-25T19:13:29.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.380+0000 7f38a2ffd640 1 -- 172.21.15.28:0/2468746042 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+17017 (secure 0 0 0) 0x7f38a405f4a0 con 0x7f38ac06d580 2024-04-25T19:13:29.382 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:29.382 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":49,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:27.025837+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f388c0744b0 msgr2=0x7f388c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f388c0744b0 0x7f388c0769c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f38ac1549e0 tx=0x7f3894008040 comp rx=0 tx=0).stop 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06d580 msgr2=0x7f38ac09b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06d580 0x7f38ac09b540 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f38a400b3c0 tx=0x7f38a400b880 comp rx=0 tx=0).stop 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 shutdown_connections 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f388c0744b0 0x7f388c0769c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f38ac15f180 0x7f38ac1607c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38ac06f1e0 0x7f38ac1ee470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 --2- 172.21.15.28:0/2468746042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38ac06d580 0x7f38ac09b540 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 >> 172.21.15.28:0/2468746042 conn(0x7f38ac0732b0 msgr2=0x7f38ac0b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:29.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 shutdown_connections 2024-04-25T19:13:29.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.382+0000 7f38b0f38640 1 -- 172.21.15.28:0/2468746042 wait complete. 2024-04-25T19:13:29.395 INFO:tasks.cephfs.filesystem:Destroying file system recovery_fs and related pools 2024-04-25T19:13:29.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.505+0000 7f5876b72640 1 Processor -- start 2024-04-25T19:13:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.505+0000 7f5876b72640 1 -- start start 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.506+0000 7f5876b72640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f587009cb20 0x7f58700b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f5876b72640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5870163850 0x7f58700ce010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f5876b72640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58700d1d30 con 0x7f58700ce550 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f5876b72640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f58700d2620 con 0x7f5870163850 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f5876b72640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f587006ebb0 con 0x7f587009cb20 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f586ffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5870163850 0x7f58700ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f586ffff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5870163850 0x7f58700ce010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36000/0 (socket says 172.21.15.28:36000) 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f586ffff640 1 -- 172.21.15.28:0/3644414807 learned_addr learned my addr 172.21.15.28:0/3644414807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f587490e640 1 --2- 172.21.15.28:0/3644414807 >> v2:172.21.15.53:3300/0 conn(0x7f587009cb20 0x7f58700b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f586ffff640 1 -- 172.21.15.28:0/3644414807 >> v2:172.21.15.53:3300/0 conn(0x7f587009cb20 msgr2=0x7f58700b91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f586ffff640 1 --2- 172.21.15.28:0/3644414807 >> v2:172.21.15.53:3300/0 conn(0x7f587009cb20 0x7f58700b91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f586ffff640 1 -- 172.21.15.28:0/3644414807 >> v1:172.21.15.53:6789/0 conn(0x7f58700ce550 legacy=0x7f58700d2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.507+0000 7f586ffff640 1 -- 172.21.15.28:0/3644414807 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f587006e5f0 con 0x7f5870163850 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.508+0000 7f586ffff640 1 --2- 172.21.15.28:0/3644414807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5870163850 0x7f58700ce010 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f587008cde0 tx=0x7f586402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4dc38e258c7d437 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.508+0000 7f586f7fe640 1 -- 172.21.15.28:0/3644414807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864037070 con 0x7f5870163850 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.508+0000 7f586f7fe640 1 -- 172.21.15.28:0/3644414807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5864032440 con 0x7f5870163850 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.508+0000 7f586f7fe640 1 -- 172.21.15.28:0/3644414807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864032810 con 0x7f5870163850 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.509+0000 7f5876b72640 1 -- 172.21.15.28:0/3644414807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5870163850 msgr2=0x7f58700ce010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.509+0000 7f5876b72640 1 --2- 172.21.15.28:0/3644414807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5870163850 0x7f58700ce010 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f587008cde0 tx=0x7f586402a270 comp rx=0 tx=0).stop 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.509+0000 7f5876b72640 1 -- 172.21.15.28:0/3644414807 shutdown_connections 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.509+0000 7f5876b72640 1 --2- 172.21.15.28:0/3644414807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5870163850 0x7f58700ce010 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.509+0000 7f5876b72640 1 --2- 172.21.15.28:0/3644414807 >> v2:172.21.15.53:3300/0 conn(0x7f587009cb20 0x7f58700b91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.509+0000 7f5876b72640 1 -- 172.21.15.28:0/3644414807 >> 172.21.15.28:0/3644414807 conn(0x7f58700b54b0 msgr2=0x7f58700b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.509+0000 7f5876b72640 1 -- 172.21.15.28:0/3644414807 shutdown_connections 2024-04-25T19:13:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.509+0000 7f5876b72640 1 -- 172.21.15.28:0/3644414807 wait complete. 2024-04-25T19:13:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.510+0000 7f5876b72640 1 Processor -- start 2024-04-25T19:13:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.511+0000 7f5876b72640 1 -- start start 2024-04-25T19:13:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f5876b72640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f587009cb20 0x7f587009d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f5876b72640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58700ce550 0x7f58701528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f5876b72640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5870163850 0x7f587014f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f5876b72640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58700d22b0 con 0x7f58700ce550 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f5876b72640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f58700d1fd0 con 0x7f587009cb20 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f5876b72640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5870163c60 con 0x7f5870163850 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f586ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58700ce550 0x7f58701528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f586ffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58700ce550 0x7f58701528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36004/0 (socket says 172.21.15.28:36004) 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f586ffff640 1 -- 172.21.15.28:0/2456563006 learned_addr learned my addr 172.21.15.28:0/2456563006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f587490e640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f587009cb20 0x7f587009d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f587510f640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5870163850 0x7f587014f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f586ffff640 1 -- 172.21.15.28:0/2456563006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5870163850 msgr2=0x7f587014f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f586ffff640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5870163850 0x7f587014f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f586ffff640 1 -- 172.21.15.28:0/2456563006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f587009cb20 msgr2=0x7f587009d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f586ffff640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f587009cb20 0x7f587009d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.512+0000 7f586ffff640 1 -- 172.21.15.28:0/2456563006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5870152e20 con 0x7f58700ce550 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.513+0000 7f586ffff640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58700ce550 0x7f58701528e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f586402a2b0 tx=0x7f58640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.513+0000 7f586d7fa640 1 -- 172.21.15.28:0/2456563006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864041020 con 0x7f58700ce550 2024-04-25T19:13:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.513+0000 7f586d7fa640 1 -- 172.21.15.28:0/2456563006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58640306e0 con 0x7f58700ce550 2024-04-25T19:13:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.513+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5870150300 con 0x7f58700ce550 2024-04-25T19:13:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.513+0000 7f586d7fa640 1 -- 172.21.15.28:0/2456563006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864037040 con 0x7f58700ce550 2024-04-25T19:13:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.513+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58701508c0 con 0x7f58700ce550 2024-04-25T19:13:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.514+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5870150070 con 0x7f58700ce550 2024-04-25T19:13:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.515+0000 7f586d7fa640 1 -- 172.21.15.28:0/2456563006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5864044020 con 0x7f58700ce550 2024-04-25T19:13:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.515+0000 7f586d7fa640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f585c0744b0 0x7f585c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.515+0000 7f586d7fa640 1 -- 172.21.15.28:0/2456563006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7f58640b5c60 con 0x7f58700ce550 2024-04-25T19:13:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.515+0000 7f587490e640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f585c0744b0 0x7f585c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.516+0000 7f587490e640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f585c0744b0 0x7f585c0769c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5860006fd0 tx=0x7f5860008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.520+0000 7f586d7fa640 1 -- 172.21.15.28:0/2456563006 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f586404b030 con 0x7f58700ce550 2024-04-25T19:13:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.741+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5870050740 con 0x7f58700ce550 2024-04-25T19:13:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.742+0000 7f586d7fa640 1 -- 172.21.15.28:0/2456563006 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5184 (secure 0 0 0) 0x7f5870152e20 con 0x7f58700ce550 2024-04-25T19:13:29.744 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:29.744 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:29.745 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-04-25T19:13:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f585c0744b0 msgr2=0x7f585c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f585c0744b0 0x7f585c0769c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5860006fd0 tx=0x7f5860008040 comp rx=0 tx=0).stop 2024-04-25T19:13:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58700ce550 msgr2=0x7f58701528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58700ce550 0x7f58701528e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f586402a2b0 tx=0x7f58640046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 shutdown_connections 2024-04-25T19:13:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f585c0744b0 0x7f585c0769c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5870163850 0x7f587014f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58700ce550 0x7f58701528e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 --2- 172.21.15.28:0/2456563006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f587009cb20 0x7f587009d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.744+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 >> 172.21.15.28:0/2456563006 conn(0x7f58700b54b0 msgr2=0x7f58700b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:29.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.745+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 shutdown_connections 2024-04-25T19:13:29.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.745+0000 7f5876b72640 1 -- 172.21.15.28:0/2456563006 wait complete. 2024-04-25T19:13:29.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:13:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.868+0000 7fbaf27b2640 1 Processor -- start 2024-04-25T19:13:29.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.868+0000 7fbaf27b2640 1 -- start start 2024-04-25T19:13:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.869+0000 7fbaf27b2640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbaec09caf0 0x7fbaec0d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.869+0000 7fbaf27b2640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbaec09cf00 0x7fbaec0ce720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.869+0000 7fbaf27b2640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaec0d1ce0 con 0x7fbaec0d2670 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.869+0000 7fbaf27b2640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fbaec0b8fa0 con 0x7fbaec09caf0 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.869+0000 7fbaf27b2640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fbaec0cf8d0 con 0x7fbaec09cf00 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.869+0000 7fbaebfff640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbaec09caf0 0x7fbaec0d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.869+0000 7fbaeb7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbaec0d2670 0x7fbaec0b71d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39898/0 (socket says 172.21.15.28:39898) 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.869+0000 7fbaeb7fe640 1 -- 172.21.15.28:0/1275720440 learned_addr learned my addr 172.21.15.28:0/1275720440 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.870+0000 7fbaf0d4f640 1 --2- 172.21.15.28:0/1275720440 >> v2:172.21.15.53:3300/0 conn(0x7fbaec09cf00 0x7fbaec0ce720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.870+0000 7fbaebfff640 1 -- 172.21.15.28:0/1275720440 >> v2:172.21.15.53:3300/0 conn(0x7fbaec09cf00 msgr2=0x7fbaec0ce720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.870+0000 7fbaebfff640 1 --2- 172.21.15.28:0/1275720440 >> v2:172.21.15.53:3300/0 conn(0x7fbaec09cf00 0x7fbaec0ce720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.870+0000 7fbaebfff640 1 -- 172.21.15.28:0/1275720440 >> v1:172.21.15.28:6789/0 conn(0x7fbaec0d2670 legacy=0x7fbaec0b71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.870+0000 7fbaeaffd640 1 -- 172.21.15.28:0/1275720440 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575191634 0 0) 0x7fbaec0d1ce0 con 0x7fbaec0d2670 2024-04-25T19:13:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.870+0000 7fbaebfff640 1 -- 172.21.15.28:0/1275720440 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec0cfbb0 con 0x7fbaec09caf0 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaebfff640 1 --2- 172.21.15.28:0/1275720440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09caf0 0x7fbaec0d2130 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbaec074cf0 tx=0x7fbadc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50df055d8037bf14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaeaffd640 1 -- 172.21.15.28:0/1275720440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc037070 con 0x7fbaec09caf0 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaeaffd640 1 -- 172.21.15.28:0/1275720440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbadc032440 con 0x7fbaec09caf0 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaeaffd640 1 -- 172.21.15.28:0/1275720440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbadc02ae60 con 0x7fbaec09caf0 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1275720440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09caf0 msgr2=0x7fbaec0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1275720440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09caf0 0x7fbaec0d2130 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbaec074cf0 tx=0x7fbadc02a110 comp rx=0 tx=0).stop 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1275720440 shutdown_connections 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1275720440 >> v2:172.21.15.53:3300/0 conn(0x7fbaec09cf00 0x7fbaec0ce720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1275720440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09caf0 0x7fbaec0d2130 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1275720440 >> 172.21.15.28:0/1275720440 conn(0x7fbaec0d03f0 msgr2=0x7fbaec0d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1275720440 shutdown_connections 2024-04-25T19:13:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.871+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1275720440 wait complete. 2024-04-25T19:13:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.873+0000 7fbaf27b2640 1 Processor -- start 2024-04-25T19:13:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.874+0000 7fbaf27b2640 1 -- start start 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaf27b2640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbaec09caf0 0x7fbaec09d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaf27b2640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09cf00 0x7fbaec1e3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaf27b2640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbaec0d2670 0x7fbaec0a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaf27b2640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec0b9430 con 0x7fbaec09cf00 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaf27b2640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fbaec0b78e0 con 0x7fbaec09caf0 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaf27b2640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fbaec0d4790 con 0x7fbaec0d2670 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09cf00 0x7fbaec1e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaeb7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09cf00 0x7fbaec1e3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36022/0 (socket says 172.21.15.28:36022) 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaeb7fe640 1 -- 172.21.15.28:0/1782784210 learned_addr learned my addr 172.21.15.28:0/1782784210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaebfff640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbaec09caf0 0x7fbaec09d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaf0d4f640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbaec0d2670 0x7fbaec0a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaeb7fe640 1 -- 172.21.15.28:0/1782784210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbaec0d2670 msgr2=0x7fbaec0a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaeb7fe640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbaec0d2670 0x7fbaec0a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaeb7fe640 1 -- 172.21.15.28:0/1782784210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbaec09caf0 msgr2=0x7fbaec09d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaeb7fe640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbaec09caf0 0x7fbaec09d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.875+0000 7fbaeb7fe640 1 -- 172.21.15.28:0/1782784210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaec0c8620 con 0x7fbaec09cf00 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.876+0000 7fbaeb7fe640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09cf00 0x7fbaec1e3210 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbae000c430 tx=0x7fbae000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.876+0000 7fbae8ff9640 1 -- 172.21.15.28:0/1782784210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0017070 con 0x7fbaec09cf00 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.876+0000 7fbae8ff9640 1 -- 172.21.15.28:0/1782784210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbae0005ce0 con 0x7fbaec09cf00 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.876+0000 7fbae8ff9640 1 -- 172.21.15.28:0/1782784210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbae0007310 con 0x7fbaec09cf00 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.876+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaec09e090 con 0x7fbaec09cf00 2024-04-25T19:13:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.876+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaec09e650 con 0x7fbaec09cf00 2024-04-25T19:13:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.877+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaec050740 con 0x7fbaec09cf00 2024-04-25T19:13:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.878+0000 7fbae8ff9640 1 -- 172.21.15.28:0/1782784210 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbae001a070 con 0x7fbaec09cf00 2024-04-25T19:13:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.878+0000 7fbae8ff9640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbad80744e0 0x7fbad80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.878+0000 7fbae8ff9640 1 -- 172.21.15.28:0/1782784210 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7fbae0094780 con 0x7fbaec09cf00 2024-04-25T19:13:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.878+0000 7fbaebfff640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbad80744e0 0x7fbad80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:29.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.879+0000 7fbaebfff640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbad80744e0 0x7fbad80769f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbadc034000 tx=0x7fbadc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:29.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:29.883+0000 7fbae8ff9640 1 -- 172.21.15.28:0/1782784210 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbae005e6b0 con 0x7fbaec09cf00 2024-04-25T19:13:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.105+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbaec0d2340 con 0x7fbaec09cf00 2024-04-25T19:13:30.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.106+0000 7fbae8ff9640 1 -- 172.21.15.28:0/1782784210 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5184 (secure 0 0 0) 0x7fbae00629b0 con 0x7fbaec09cf00 2024-04-25T19:13:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:30.108 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T19:13:03.631282+0000","modified":"2024-04-25T19:13:13.979864+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5589},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5589":{"gid":5589,"name":"c","rank":0,"incarnation":27,"state":"up:active","state_seq":6,"addr":"172.21.15.28:6837/1842070529","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":1842070529},{"type":"v1","addr":"172.21.15.28:6837","nonce":1842070529}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":7,"enabled":true,"fs_name":"recovery_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5589,"qdb_cluster":[5589]},"id":3}]} 2024-04-25T19:13:30.108 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 32 2024-04-25T19:13:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbad80744e0 msgr2=0x7fbad80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbad80744e0 0x7fbad80769f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbadc034000 tx=0x7fbadc02e000 comp rx=0 tx=0).stop 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09cf00 msgr2=0x7fbaec1e3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09cf00 0x7fbaec1e3210 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbae000c430 tx=0x7fbae000c8f0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 shutdown_connections 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fbad80744e0 0x7fbad80769f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fbaec0d2670 0x7fbaec0a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbaec09cf00 0x7fbaec1e3210 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 --2- 172.21.15.28:0/1782784210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbaec09caf0 0x7fbaec09d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 >> 172.21.15.28:0/1782784210 conn(0x7fbaec0d03f0 msgr2=0x7fbaec072da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 shutdown_connections 2024-04-25T19:13:30.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.108+0000 7fbaf27b2640 1 -- 172.21.15.28:0/1782784210 wait complete. 2024-04-25T19:13:30.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.231+0000 7f86ecee0640 1 Processor -- start 2024-04-25T19:13:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.231+0000 7f86ecee0640 1 -- start start 2024-04-25T19:13:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86ecee0640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f86e80b4e30 0x7f86e80bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86ecee0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b9470 0x7f86e80b9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86ecee0640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e80bbb90 con 0x7f86e80b7650 2024-04-25T19:13:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86ecee0640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f86e80d0b50 con 0x7f86e80b4e30 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86ecee0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f86e808e3a0 con 0x7f86e80b9470 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86e6d9d640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f86e80b7650 0x7f86e80b7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39908/0 (socket says 172.21.15.28:39908) 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86e6d9d640 1 -- 172.21.15.28:0/1607849620 learned_addr learned my addr 172.21.15.28:0/1607849620 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86e659c640 1 --2- 172.21.15.28:0/1607849620 >> v2:172.21.15.53:3300/0 conn(0x7f86e80b4e30 0x7f86e80bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86dffff640 1 --2- 172.21.15.28:0/1607849620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b9470 0x7f86e80b9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86e5d9b640 1 -- 172.21.15.28:0/1607849620 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 592600946 0 0) 0x7f86e0001490 con 0x7f86e80b7650 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86e5d9b640 1 -- 172.21.15.28:0/1607849620 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86d4003520 con 0x7f86e80b7650 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86dffff640 1 -- 172.21.15.28:0/1607849620 >> v2:172.21.15.53:3300/0 conn(0x7f86e80b4e30 msgr2=0x7f86e80bb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86dffff640 1 --2- 172.21.15.28:0/1607849620 >> v2:172.21.15.53:3300/0 conn(0x7f86e80b4e30 0x7f86e80bb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86dffff640 1 -- 172.21.15.28:0/1607849620 >> v1:172.21.15.28:6789/0 conn(0x7f86e80b7650 legacy=0x7f86e80b7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86dffff640 1 -- 172.21.15.28:0/1607849620 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e8164d60 con 0x7f86e80b9470 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.233+0000 7f86e659c640 1 --2- 172.21.15.28:0/1607849620 >> v2:172.21.15.53:3300/0 conn(0x7f86e80b4e30 0x7f86e80bb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.234+0000 7f86dffff640 1 --2- 172.21.15.28:0/1607849620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b9470 0x7f86e80b9880 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f86e80d0cd0 tx=0x7f86d002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae0b29e405ac879f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.234+0000 7f86e5d9b640 1 -- 172.21.15.28:0/1607849620 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0037070 con 0x7f86e80b9470 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.234+0000 7f86e5d9b640 1 -- 172.21.15.28:0/1607849620 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d002ae50 con 0x7f86e80b9470 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.234+0000 7f86e5d9b640 1 -- 172.21.15.28:0/1607849620 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0032640 con 0x7f86e80b9470 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.235+0000 7f86ecee0640 1 -- 172.21.15.28:0/1607849620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b9470 msgr2=0x7f86e80b9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.235+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1607849620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b9470 0x7f86e80b9880 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f86e80d0cd0 tx=0x7f86d002a100 comp rx=0 tx=0).stop 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.235+0000 7f86ecee0640 1 -- 172.21.15.28:0/1607849620 shutdown_connections 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.235+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1607849620 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b9470 0x7f86e80b9880 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.235+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1607849620 >> v2:172.21.15.53:3300/0 conn(0x7f86e80b4e30 0x7f86e80bb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.235+0000 7f86ecee0640 1 -- 172.21.15.28:0/1607849620 >> 172.21.15.28:0/1607849620 conn(0x7f86e80d2650 msgr2=0x7f86e80d2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:30.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.235+0000 7f86ecee0640 1 -- 172.21.15.28:0/1607849620 shutdown_connections 2024-04-25T19:13:30.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.235+0000 7f86ecee0640 1 -- 172.21.15.28:0/1607849620 wait complete. 2024-04-25T19:13:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.237+0000 7f86ecee0640 1 Processor -- start 2024-04-25T19:13:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.237+0000 7f86ecee0640 1 -- start start 2024-04-25T19:13:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86ecee0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b4e30 0x7f86e815a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86ecee0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e80b7650 0x7f86e8163030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86ecee0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86e80b9470 0x7f86e80ba470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86ecee0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86e80d07e0 con 0x7f86e80b7650 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86ecee0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f86e80d0500 con 0x7f86e80b9470 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86ecee0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f86e81649c0 con 0x7f86e80b4e30 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e80b7650 0x7f86e8163030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86dffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e80b7650 0x7f86e8163030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36038/0 (socket says 172.21.15.28:36038) 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.238+0000 7f86dffff640 1 -- 172.21.15.28:0/1638239562 learned_addr learned my addr 172.21.15.28:0/1638239562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86e659c640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b4e30 0x7f86e815a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86e6d9d640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86e80b9470 0x7f86e80ba470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86dffff640 1 -- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b4e30 msgr2=0x7f86e815a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86dffff640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b4e30 0x7f86e815a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86dffff640 1 -- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86e80b9470 msgr2=0x7f86e80ba470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86dffff640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86e80b9470 0x7f86e80ba470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86dffff640 1 -- 172.21.15.28:0/1638239562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e81fb040 con 0x7f86e80b7650 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86e6d9d640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86e80b9470 0x7f86e80ba470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86e659c640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b4e30 0x7f86e815a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86dffff640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e80b7650 0x7f86e8163030 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f86d0000c00 tx=0x7f86d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86deffd640 1 -- 172.21.15.28:0/1638239562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0041030 con 0x7f86e80b7650 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86deffd640 1 -- 172.21.15.28:0/1638239562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d002ac50 con 0x7f86e80b7650 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e81fa430 con 0x7f86e80b7650 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86deffd640 1 -- 172.21.15.28:0/1638239562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0037040 con 0x7f86e80b7650 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.239+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e81fa9f0 con 0x7f86e80b7650 2024-04-25T19:13:30.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.240+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e81fa1a0 con 0x7f86e80b7650 2024-04-25T19:13:30.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.241+0000 7f86deffd640 1 -- 172.21.15.28:0/1638239562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f86d0044070 con 0x7f86e80b7650 2024-04-25T19:13:30.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.241+0000 7f86deffd640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86c80744b0 0x7f86c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.241+0000 7f86deffd640 1 -- 172.21.15.28:0/1638239562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7f86d00b5a60 con 0x7f86e80b7650 2024-04-25T19:13:30.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.241+0000 7f86e659c640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86c80744b0 0x7f86c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.242+0000 7f86e659c640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86c80744b0 0x7f86c80769c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f86e80d2620 tx=0x7f86d8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.246+0000 7f86deffd640 1 -- 172.21.15.28:0/1638239562 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f86d007ffe0 con 0x7f86e80b7650 2024-04-25T19:13:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.431+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86e80b4640 con 0x7f86e80b7650 2024-04-25T19:13:30.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.432+0000 7f86deffd640 1 -- 172.21.15.28:0/1638239562 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) ==== 74+0+17017 (secure 0 0 0) 0x7f86e80b4640 con 0x7f86e80b7650 2024-04-25T19:13:30.434 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:30.434 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":49,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:27.025837+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T19:12:43.014197+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"recovery_fs_metadata","create_time":"2024-04-25T19:13:02.679639+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"recovery_fs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.24","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:30.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86c80744b0 msgr2=0x7f86c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86c80744b0 0x7f86c80769c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f86e80d2620 tx=0x7f86d8008040 comp rx=0 tx=0).stop 2024-04-25T19:13:30.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e80b7650 msgr2=0x7f86e8163030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e80b7650 0x7f86e8163030 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f86d0000c00 tx=0x7f86d00046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 shutdown_connections 2024-04-25T19:13:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f86c80744b0 0x7f86c80769c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86e80b9470 0x7f86e80ba470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86e80b7650 0x7f86e8163030 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 --2- 172.21.15.28:0/1638239562 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f86e80b4e30 0x7f86e815a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 >> 172.21.15.28:0/1638239562 conn(0x7f86e80d2650 msgr2=0x7f86e80d5b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 shutdown_connections 2024-04-25T19:13:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.434+0000 7f86ecee0640 1 -- 172.21.15.28:0/1638239562 wait complete. 2024-04-25T19:13:30.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail recovery_fs 2024-04-25T19:13:30.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.559+0000 7f0fd908c640 1 Processor -- start 2024-04-25T19:13:30.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.559+0000 7f0fd908c640 1 -- start start 2024-04-25T19:13:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd908c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0fd40b30e0 0x7f0fd40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd908c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd406d1f0 0x7f0fd406d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd908c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fd40b3650 con 0x7f0fd409b540 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd908c640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f0fd406c710 con 0x7f0fd40b30e0 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd908c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd406c9f0 con 0x7f0fd406d1f0 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd259c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0fd409b540 0x7f0fd409b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39920/0 (socket says 172.21.15.28:39920) 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd259c640 1 -- 172.21.15.28:0/660585355 learned_addr learned my addr 172.21.15.28:0/660585355 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd359e640 1 --2- 172.21.15.28:0/660585355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd406d1f0 0x7f0fd406d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd2d9d640 1 --2- 172.21.15.28:0/660585355 >> v2:172.21.15.53:3300/0 conn(0x7f0fd40b30e0 0x7f0fd40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd1d9b640 1 -- 172.21.15.28:0/660585355 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1842723296 0 0) 0x7f0fd40b3650 con 0x7f0fd409b540 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.560+0000 7f0fd1d9b640 1 -- 172.21.15.28:0/660585355 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0fbc003520 con 0x7f0fd409b540 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.561+0000 7f0fd1d9b640 1 -- 172.21.15.28:0/660585355 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1365824183 0 0) 0x7f0fbc003520 con 0x7f0fd409b540 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.561+0000 7f0fd359e640 1 -- 172.21.15.28:0/660585355 >> v2:172.21.15.53:3300/0 conn(0x7f0fd40b30e0 msgr2=0x7f0fd40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.561+0000 7f0fd359e640 1 --2- 172.21.15.28:0/660585355 >> v2:172.21.15.53:3300/0 conn(0x7f0fd40b30e0 0x7f0fd40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.561+0000 7f0fd359e640 1 -- 172.21.15.28:0/660585355 >> v1:172.21.15.28:6789/0 conn(0x7f0fd409b540 legacy=0x7f0fd409b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.561+0000 7f0fd359e640 1 -- 172.21.15.28:0/660585355 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd406ccd0 con 0x7f0fd406d1f0 2024-04-25T19:13:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.561+0000 7f0fd2d9d640 1 --2- 172.21.15.28:0/660585355 >> v2:172.21.15.53:3300/0 conn(0x7f0fd40b30e0 0x7f0fd40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.562+0000 7f0fd359e640 1 --2- 172.21.15.28:0/660585355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd406d1f0 0x7f0fd406d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009d90 tx=0x7f0fc402a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=615714c832d005e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.562+0000 7f0fd1d9b640 1 -- 172.21.15.28:0/660585355 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8001690 con 0x7f0fd406d1f0 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.562+0000 7f0fd1d9b640 1 -- 172.21.15.28:0/660585355 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc4007540 con 0x7f0fd406d1f0 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.562+0000 7f0fd1d9b640 1 -- 172.21.15.28:0/660585355 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc40078f0 con 0x7f0fd406d1f0 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.563+0000 7f0fd908c640 1 -- 172.21.15.28:0/660585355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd406d1f0 msgr2=0x7f0fd406d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.563+0000 7f0fd908c640 1 --2- 172.21.15.28:0/660585355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd406d1f0 0x7f0fd406d600 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009d90 tx=0x7f0fc402a560 comp rx=0 tx=0).stop 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.563+0000 7f0fd908c640 1 -- 172.21.15.28:0/660585355 shutdown_connections 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.563+0000 7f0fd908c640 1 --2- 172.21.15.28:0/660585355 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd406d1f0 0x7f0fd406d600 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.563+0000 7f0fd908c640 1 --2- 172.21.15.28:0/660585355 >> v2:172.21.15.53:3300/0 conn(0x7f0fd40b30e0 0x7f0fd40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.563+0000 7f0fd908c640 1 -- 172.21.15.28:0/660585355 >> 172.21.15.28:0/660585355 conn(0x7f0fd40b7620 msgr2=0x7f0fd40b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:30.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.563+0000 7f0fd908c640 1 -- 172.21.15.28:0/660585355 shutdown_connections 2024-04-25T19:13:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.563+0000 7f0fd908c640 1 -- 172.21.15.28:0/660585355 wait complete. 2024-04-25T19:13:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.565+0000 7f0fd908c640 1 Processor -- start 2024-04-25T19:13:30.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.565+0000 7f0fd908c640 1 -- start start 2024-04-25T19:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd908c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd406d1f0 0x7f0fd4156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd908c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd409b540 0x7f0fd415f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd908c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd40b30e0 0x7f0fd40b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd908c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd4161ac0 con 0x7f0fd40b30e0 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd908c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f0fd4161940 con 0x7f0fd406d1f0 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd908c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f0fd406db40 con 0x7f0fd409b540 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd40b30e0 0x7f0fd40b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd359e640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd40b30e0 0x7f0fd40b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36040/0 (socket says 172.21.15.28:36040) 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd359e640 1 -- 172.21.15.28:0/433034727 learned_addr learned my addr 172.21.15.28:0/433034727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.566+0000 7f0fd2d9d640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd406d1f0 0x7f0fd4156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd259c640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd409b540 0x7f0fd415f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd359e640 1 -- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd409b540 msgr2=0x7f0fd415f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd359e640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd409b540 0x7f0fd415f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd359e640 1 -- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd406d1f0 msgr2=0x7f0fd4156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd359e640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd406d1f0 0x7f0fd4156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd359e640 1 -- 172.21.15.28:0/433034727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fd415fe40 con 0x7f0fd40b30e0 2024-04-25T19:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd259c640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd409b540 0x7f0fd415f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd2d9d640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd406d1f0 0x7f0fd4156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd359e640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd40b30e0 0x7f0fd40b0f90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0fc402ab50 tx=0x7f0fc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fbb7fe640 1 -- 172.21.15.28:0/433034727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc403f070 con 0x7f0fd40b30e0 2024-04-25T19:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fbb7fe640 1 -- 172.21.15.28:0/433034727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fc4031c40 con 0x7f0fd40b30e0 2024-04-25T19:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fd4160120 con 0x7f0fd40b30e0 2024-04-25T19:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fbb7fe640 1 -- 172.21.15.28:0/433034727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4009450 con 0x7f0fd40b30e0 2024-04-25T19:13:30.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.567+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fd41606e0 con 0x7f0fd40b30e0 2024-04-25T19:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.568+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fd4050740 con 0x7f0fd40b30e0 2024-04-25T19:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.569+0000 7f0fbb7fe640 1 -- 172.21.15.28:0/433034727 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0fc4043410 con 0x7f0fd40b30e0 2024-04-25T19:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.569+0000 7f0fbb7fe640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0fc00744e0 0x7f0fc00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.569+0000 7f0fbb7fe640 1 -- 172.21.15.28:0/433034727 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 6231+0+0 (secure 0 0 0) 0x7f0fc40b6790 con 0x7f0fd40b30e0 2024-04-25T19:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.569+0000 7f0fd2d9d640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0fc00744e0 0x7f0fc00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.570+0000 7f0fd2d9d640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0fc00744e0 0x7f0fc00769f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40b1f90 tx=0x7f0fcc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.574+0000 7f0fbb7fe640 1 -- 172.21.15.28:0/433034727 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0fc4080c80 con 0x7f0fd40b30e0 2024-04-25T19:13:30.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.800+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "recovery_fs"} v 0) -- 0x7f0fd406ea30 con 0x7f0fd40b30e0 2024-04-25T19:13:30.803 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-04-25T19:13:30.801+0000 7fa80f29e640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T19:13:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.802+0000 7f0fbb7fe640 1 -- 172.21.15.28:0/433034727 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "recovery_fs"}]=0 recovery_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) ==== 171+0+0 (secure 0 0 0) 0x7f0fc4080580 con 0x7f0fd40b30e0 2024-04-25T19:13:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:recovery_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T19:13:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0fc00744e0 msgr2=0x7f0fc00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0fc00744e0 0x7f0fc00769f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0fd40b1f90 tx=0x7f0fcc008040 comp rx=0 tx=0).stop 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd40b30e0 msgr2=0x7f0fd40b0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd40b30e0 0x7f0fd40b0f90 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f0fc402ab50 tx=0x7f0fc4004a40 comp rx=0 tx=0).stop 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 shutdown_connections 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f0fc00744e0 0x7f0fc00769f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0fd40b30e0 0x7f0fd40b0f90 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f0fd409b540 0x7f0fd415f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 --2- 172.21.15.28:0/433034727 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd406d1f0 0x7f0fd4156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 >> 172.21.15.28:0/433034727 conn(0x7f0fd40b7620 msgr2=0x7f0fd40704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 shutdown_connections 2024-04-25T19:13:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.806+0000 7f0fd908c640 1 -- 172.21.15.28:0/433034727 wait complete. 2024-04-25T19:13:30.810 INFO:tasks.ceph.mds.c.smithi028.stderr: -5114> 2024-04-25T19:13:10.979+0000 7f95c5b58640 -1 mds.0.openfiles _load_finish got (2) No such file or directory 2024-04-25T19:13:30.811 INFO:tasks.ceph.mds.c.smithi028.stderr: -3449> 2024-04-25T19:13:16.008+0000 7f95c5b58640 -1 log_channel(cluster) log [ERR] : bad backtrace on directory inode 0x10000000000 2024-04-25T19:13:30.811 INFO:tasks.ceph.mds.c.smithi028.stderr: -3279> 2024-04-25T19:13:16.009+0000 7f95c5b58640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /subdir/ auth v1 pv4 ap=3 DIRTYPARENT f(v0 1=1+0) n() (inest lock dirty) (ifile lock->sync w=1 flushing) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55bea449d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x10000000000:[<0x1/subdir v1>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=1+0)","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 1=1+0)","memory_value.rstat":"n()","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-04-25T19:13:30.811 INFO:tasks.ceph.mds.c.smithi028.stderr: -3071> 2024-04-25T19:13:16.012+0000 7f95c5b58640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /subdir/sixmegs auth v1 ap=1 DIRTYPARENT s=6291456 n() | dirtyparent=1 authpin=1 scrubqueue=0 0x55bea4514100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(6)0x10000000001:[<0x10000000000/sixmegs v10>,<0x1/subdir v12>]//[]","memoryvalue":"(6)0x10000000001:[<0x10000000000/sixmegs v1>,<0x1/subdir v4>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-04-25T19:13:30.811 INFO:tasks.ceph.mds.c.smithi028.stderr: -2879> 2024-04-25T19:13:16.014+0000 7f95c5b58640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1 [...2,head] / auth v4 pv6 ap=5 snaprealm=0x55bea4498480 DIRTYPARENT f(v1 1=0+1) n(v0 1=0+1)/n() (inest mix->lock(2) w=1 flushing) (ifile mix->sync w=1 flushing) | dirtyscattered=2 lock=2 dirfrag=1 openingsnapparents=0 dirtyparent=1 dirty=1 waiter=1 authpin=1 scrubqueue=0 0x55bea449cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x1:[]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 1=0+1)","ondisk_value.rstat":"n(v0 2=0+2)","memory_value.dirstat":"f(v1 1=0+1)","memory_value.rstat":"n(v0 1=0+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-61} 2024-04-25T19:13:30.812 INFO:tasks.ceph.mds.c.smithi028.stderr: -2741> 2024-04-25T19:13:16.922+0000 7f95c9b60640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T19:13:30.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm recovery_fs --yes-i-really-mean-it 2024-04-25T19:13:30.867 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-04-25T19:13:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.934+0000 7fb14d114640 1 Processor -- start 2024-04-25T19:13:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.934+0000 7fb14d114640 1 -- start start 2024-04-25T19:13:30.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.935+0000 7fb14d114640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb14815f270 0x7fb1480b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb14d114640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1480b4df0 con 0x7fb1480b49e0 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb14d114640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14815f680 con 0x7fb1480ceca0 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb14d114640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb1480cf0b0 con 0x7fb14815f270 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb14659c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb1480b49e0 0x7fb1480ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50836/0 (socket says 172.21.15.28:50836) 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb14659c640 1 -- 172.21.15.28:0/1783717266 learned_addr learned my addr 172.21.15.28:0/1783717266 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb146d9d640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb14815f270 0x7fb1480b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1058083936 0 0) 0x7fb130001110 con 0x7fb1480b49e0 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb134003520 con 0x7fb1480b49e0 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094700161 0 0) 0x7fb14815f680 con 0x7fb1480ceca0 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb130001110 con 0x7fb1480ceca0 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3244972843 0 0) 0x7fb134003520 con 0x7fb1480b49e0 2024-04-25T19:13:30.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb14815f270 msgr2=0x7fb1480b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb14815f270 0x7fb1480b44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 >> v1:172.21.15.53:6789/0 conn(0x7fb1480ceca0 legacy=0x7fb1480b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1480cf2e0 con 0x7fb1480b49e0 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7fb130001840 con 0x7fb1480b49e0 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 >> v1:172.21.15.28:6789/0 conn(0x7fb1480b49e0 legacy=0x7fb1480ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb14815f270 0x7fb134003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1340040d0 0x7fb134004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb134004d40 0x7fb134005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb13400baf0 con 0x7fb1340040d0 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb13400bca0 con 0x7fb134004d40 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.936+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb13400bea0 con 0x7fb14815f270 2024-04-25T19:13:30.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb14659c640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1340040d0 0x7fb134004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb146d9d640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb14815f270 0x7fb134003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb14759e640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb134004d40 0x7fb134005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb14659c640 1 -- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb14815f270 msgr2=0x7fb134003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb14659c640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb14815f270 0x7fb134003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb14659c640 1 -- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb134004d40 msgr2=0x7fb134005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb14659c640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb134004d40 0x7fb134005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb14659c640 1 -- 172.21.15.28:0/1783717266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb134003520 con 0x7fb1340040d0 2024-04-25T19:13:30.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.937+0000 7fb14759e640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb134004d40 0x7fb134005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb14659c640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1340040d0 0x7fb134004800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb13000c4c0 tx=0x7fb134005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ab317509cbe31e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130035070 con 0x7fb1340040d0 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb13002c040 con 0x7fb1340040d0 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb145d9b640 1 -- 172.21.15.28:0/1783717266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb13000c0e0 con 0x7fb1340040d0 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb14d114640 1 -- 172.21.15.28:0/1783717266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1340040d0 msgr2=0x7fb134004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb14d114640 1 --2- 172.21.15.28:0/1783717266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1340040d0 0x7fb134004800 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb13000c4c0 tx=0x7fb134005830 comp rx=0 tx=0).stop 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb14759e640 1 -- 172.21.15.28:0/1783717266 reap_dead start 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb14d114640 1 -- 172.21.15.28:0/1783717266 shutdown_connections 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb14d114640 1 -- 172.21.15.28:0/1783717266 >> 172.21.15.28:0/1783717266 conn(0x7fb1480b7640 msgr2=0x7fb1480b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb14d114640 1 -- 172.21.15.28:0/1783717266 shutdown_connections 2024-04-25T19:13:30.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.938+0000 7fb14d114640 1 -- 172.21.15.28:0/1783717266 wait complete. 2024-04-25T19:13:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.940+0000 7fb14d114640 1 Processor -- start 2024-04-25T19:13:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.941+0000 7fb14d114640 1 -- start start 2024-04-25T19:13:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14d114640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1340040d0 0x7fb1480c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14d114640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb148158590 0x7fb1480b1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14d114640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1481589a0 0x7fb1480b1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14d114640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb14815e540 con 0x7fb148158590 2024-04-25T19:13:30.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14d114640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb14815ede0 con 0x7fb1481589a0 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14d114640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb14815ea70 con 0x7fb1340040d0 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb148158590 0x7fb1480b1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14659c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb148158590 0x7fb1480b1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55120/0 (socket says 172.21.15.28:55120) 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14659c640 1 -- 172.21.15.28:0/4085156599 learned_addr learned my addr 172.21.15.28:0/4085156599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14759e640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1481589a0 0x7fb1480b1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb146d9d640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1340040d0 0x7fb1480c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14659c640 1 -- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1340040d0 msgr2=0x7fb1480c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14659c640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1340040d0 0x7fb1480c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14659c640 1 -- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1481589a0 msgr2=0x7fb1480b1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14659c640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1481589a0 0x7fb1480b1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14659c640 1 -- 172.21.15.28:0/4085156599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1481f4e70 con 0x7fb148158590 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb14759e640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1481589a0 0x7fb1480b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.942+0000 7fb146d9d640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1340040d0 0x7fb1480c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:30.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.943+0000 7fb14659c640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb148158590 0x7fb1480b1050 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb130033040 tx=0x7fb14815ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.943+0000 7fb13f7fe640 1 -- 172.21.15.28:0/4085156599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130035040 con 0x7fb148158590 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.943+0000 7fb13f7fe640 1 -- 172.21.15.28:0/4085156599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb130007270 con 0x7fb148158590 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.943+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb148159670 con 0x7fb148158590 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.943+0000 7fb13f7fe640 1 -- 172.21.15.28:0/4085156599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130007620 con 0x7fb148158590 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.943+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1481f45d0 con 0x7fb148158590 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.944+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1480716c0 con 0x7fb148158590 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.944+0000 7fb13f7fe640 1 -- 172.21.15.28:0/4085156599 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb130030430 con 0x7fb148158590 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.945+0000 7fb13f7fe640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1280744e0 0x7fb1280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.945+0000 7fb13f7fe640 1 -- 172.21.15.28:0/4085156599 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6317+0+0 (secure 0 0 0) 0x7fb1300b5f20 con 0x7fb148158590 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.945+0000 7fb146d9d640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1280744e0 0x7fb1280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:30.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.945+0000 7fb146d9d640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1280744e0 0x7fb1280769f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb140009b80 tx=0x7fb140008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:30.950+0000 7fb13f7fe640 1 -- 172.21.15.28:0/4085156599 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1480716c0 con 0x7fb148158590 2024-04-25T19:13:31.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.174+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "recovery_fs", "yes_i_really_mean_it": true} v 0) -- 0x7fb148050740 con 0x7fb148158590 2024-04-25T19:13:31.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.806+0000 7fb13f7fe640 1 -- 172.21.15.28:0/4085156599 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "recovery_fs", "yes_i_really_mean_it": true}]=0 v34) ==== 109+0+0 (secure 0 0 0) 0x7fb1480b4df0 con 0x7fb148158590 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1280744e0 msgr2=0x7fb1280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1280744e0 0x7fb1280769f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb140009b80 tx=0x7fb140008040 comp rx=0 tx=0).stop 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb148158590 msgr2=0x7fb1480b1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb148158590 0x7fb1480b1050 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fb130033040 tx=0x7fb14815ed50 comp rx=0 tx=0).stop 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 shutdown_connections 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb1280744e0 0x7fb1280769f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1481589a0 0x7fb1480b1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb148158590 0x7fb1480b1050 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 --2- 172.21.15.28:0/4085156599 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb1340040d0 0x7fb1480c8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 >> 172.21.15.28:0/4085156599 conn(0x7fb1480b7640 msgr2=0x7fb1480b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 shutdown_connections 2024-04-25T19:13:31.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.809+0000 7fb14d114640 1 -- 172.21.15.28:0/4085156599 wait complete. 2024-04-25T19:13:31.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm recovery_fs_metadata recovery_fs_metadata --yes-i-really-really-mean-it 2024-04-25T19:13:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.941+0000 7f9f245e3640 1 Processor -- start 2024-04-25T19:13:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.941+0000 7f9f245e3640 1 -- start start 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.943+0000 7f9f245e3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.943+0000 7f9f245e3640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f1c0d12e0 con 0x7f9f1c0d5690 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.943+0000 7f9f245e3640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f1c0d15c0 con 0x7f9f1c06e5e0 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.943+0000 7f9f245e3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1c0d4bb0 con 0x7f9f1c0b47d0 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.943+0000 7f9f21b7e640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f1c0d5690 0x7f9f1c06ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50842/0 (socket says 172.21.15.28:50842) 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.943+0000 7f9f21b7e640 1 -- 172.21.15.28:0/3094290737 learned_addr learned my addr 172.21.15.28:0/3094290737 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.943+0000 7f9f2237f640 1 --2- 172.21.15.28:0/3094290737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.943+0000 7f9f2137d640 1 -- 172.21.15.28:0/3094290737 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3672214364 0 0) 0x7f9f1c0d12e0 con 0x7f9f1c0d5690 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.944+0000 7f9f2137d640 1 -- 172.21.15.28:0/3094290737 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f04003520 con 0x7f9f1c0d5690 2024-04-25T19:13:31.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.944+0000 7f9f2137d640 1 -- 172.21.15.28:0/3094290737 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1209568035 0 0) 0x7f9f1c0d15c0 con 0x7f9f1c06e5e0 2024-04-25T19:13:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.944+0000 7f9f2237f640 1 -- 172.21.15.28:0/3094290737 >> v1:172.21.15.53:6789/0 conn(0x7f9f1c06e5e0 legacy=0x7f9f1c0b9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.944+0000 7f9f2237f640 1 -- 172.21.15.28:0/3094290737 >> v1:172.21.15.28:6789/0 conn(0x7f9f1c0d5690 legacy=0x7f9f1c06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:31.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.944+0000 7f9f2237f640 1 -- 172.21.15.28:0/3094290737 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1c0d4e90 con 0x7f9f1c0b47d0 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.945+0000 7f9f2237f640 1 --2- 172.21.15.28:0/3094290737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1601f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f140097d0 tx=0x7f9f1402a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69674fc655b0d40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.945+0000 7f9f2137d640 1 -- 172.21.15.28:0/3094290737 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f14037070 con 0x7f9f1c0b47d0 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.945+0000 7f9f2137d640 1 -- 172.21.15.28:0/3094290737 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f14032440 con 0x7f9f1c0b47d0 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.945+0000 7f9f2137d640 1 -- 172.21.15.28:0/3094290737 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140327f0 con 0x7f9f1c0b47d0 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.945+0000 7f9f245e3640 1 -- 172.21.15.28:0/3094290737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 msgr2=0x7f9f1c1601f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.945+0000 7f9f245e3640 1 --2- 172.21.15.28:0/3094290737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1601f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9f140097d0 tx=0x7f9f1402a230 comp rx=0 tx=0).stop 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.945+0000 7f9f245e3640 1 -- 172.21.15.28:0/3094290737 shutdown_connections 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.946+0000 7f9f245e3640 1 --2- 172.21.15.28:0/3094290737 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1601f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.946+0000 7f9f245e3640 1 -- 172.21.15.28:0/3094290737 >> 172.21.15.28:0/3094290737 conn(0x7f9f1c0d2c60 msgr2=0x7f9f1c0d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:31.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.946+0000 7f9f245e3640 1 -- 172.21.15.28:0/3094290737 shutdown_connections 2024-04-25T19:13:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.946+0000 7f9f245e3640 1 -- 172.21.15.28:0/3094290737 wait complete. 2024-04-25T19:13:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.947+0000 7f9f245e3640 1 Processor -- start 2024-04-25T19:13:31.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.947+0000 7f9f245e3640 1 -- start start 2024-04-25T19:13:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f245e3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1633f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f245e3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f1c0d5690 0x7f9f1c1f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:31.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f245e3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1c1609b0 0x7f9f1c15d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f245e3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c0b4d00 con 0x7f9f1c1609b0 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f245e3640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1c160730 con 0x7f9f1c0d5690 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f245e3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1c0b4e80 con 0x7f9f1c0b47d0 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1c1609b0 0x7f9f1c15d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1c1609b0 0x7f9f1c15d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55132/0 (socket says 172.21.15.28:55132) 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 -- 172.21.15.28:0/14505331 learned_addr learned my addr 172.21.15.28:0/14505331 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f2237f640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f21b7e640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f1c0d5690 0x7f9f1c1f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 -- 172.21.15.28:0/14505331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 msgr2=0x7f9f1c1633f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1633f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 -- 172.21.15.28:0/14505331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f1c0d5690 msgr2=0x7f9f1c1f61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f1c0d5690 0x7f9f1c1f61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:31.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 -- 172.21.15.28:0/14505331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1c1f6710 con 0x7f9f1c1609b0 2024-04-25T19:13:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f22b80640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1c1609b0 0x7f9f1c15d630 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00c7b0 tx=0x7f9f0c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.949+0000 7f9f1affd640 1 -- 172.21.15.28:0/14505331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c00ce60 con 0x7f9f1c1609b0 2024-04-25T19:13:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.950+0000 7f9f1affd640 1 -- 172.21.15.28:0/14505331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0c003da0 con 0x7f9f1c1609b0 2024-04-25T19:13:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.950+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f1c158cd0 con 0x7f9f1c1609b0 2024-04-25T19:13:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.950+0000 7f9f1affd640 1 -- 172.21.15.28:0/14505331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c0052b0 con 0x7f9f1c1609b0 2024-04-25T19:13:31.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.950+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f1c159290 con 0x7f9f1c1609b0 2024-04-25T19:13:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.951+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f1c163980 con 0x7f9f1c1609b0 2024-04-25T19:13:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.951+0000 7f9f1affd640 1 -- 172.21.15.28:0/14505331 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9f0c005450 con 0x7f9f1c1609b0 2024-04-25T19:13:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.952+0000 7f9f1affd640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9f080744b0 0x7f9f080769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.952+0000 7f9f1affd640 1 -- 172.21.15.28:0/14505331 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 6317+0+0 (secure 0 0 0) 0x7f9f0c095130 con 0x7f9f1c1609b0 2024-04-25T19:13:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.952+0000 7f9f2237f640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9f080744b0 0x7f9f080769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:31.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.952+0000 7f9f2237f640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9f080744b0 0x7f9f080769c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9f14034000 tx=0x7f9f1402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:31.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:31.956+0000 7f9f1affd640 1 -- 172.21.15.28:0/14505331 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f1c163980 con 0x7f9f1c1609b0 2024-04-25T19:13:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:32.142+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "recovery_fs_metadata", "pool2": "recovery_fs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9f1c06a100 con 0x7f9f1c1609b0 2024-04-25T19:13:32.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:13:32.299 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:13:33.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.045+0000 7f9f1affd640 1 -- 172.21.15.28:0/14505331 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "recovery_fs_metadata", "pool2": "recovery_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'recovery_fs_metadata' removed v51) ==== 196+0+0 (secure 0 0 0) 0x7f9f1c1f6710 con 0x7f9f1c1609b0 2024-04-25T19:13:33.049 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'recovery_fs_metadata' removed 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9f080744b0 msgr2=0x7f9f080769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9f080744b0 0x7f9f080769c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9f14034000 tx=0x7f9f1402e000 comp rx=0 tx=0).stop 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1c1609b0 msgr2=0x7f9f1c15d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1c1609b0 0x7f9f1c15d630 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c00c7b0 tx=0x7f9f0c00cc70 comp rx=0 tx=0).stop 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 shutdown_connections 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f9f080744b0 0x7f9f080769c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f1c1609b0 0x7f9f1c15d630 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f1c0d5690 0x7f9f1c1f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 --2- 172.21.15.28:0/14505331 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f9f1c0b47d0 0x7f9f1c1633f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 >> 172.21.15.28:0/14505331 conn(0x7f9f1c0d2c60 msgr2=0x7f9f1c157110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:33.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 shutdown_connections 2024-04-25T19:13:33.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.049+0000 7f9f245e3640 1 -- 172.21.15.28:0/14505331 wait complete. 2024-04-25T19:13:33.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T19:13:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.187+0000 7f5c38434640 1 Processor -- start 2024-04-25T19:13:33.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.187+0000 7f5c38434640 1 -- start start 2024-04-25T19:13:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c38434640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c300b9210 0x7f5c3006e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c38434640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c3006e9d0 0x7f5c300ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c38434640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c300cdd10 con 0x7f5c300ceec0 2024-04-25T19:13:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c38434640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5c3009cf80 con 0x7f5c300b9210 2024-04-25T19:13:33.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c38434640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3006de00 con 0x7f5c3006e9d0 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c361d0640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c300b9210 0x7f5c3006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c359cf640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c3006e9d0 0x7f5c300ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c361d0640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c300b9210 0x7f5c3006e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37052/0 (socket says 172.21.15.28:37052) 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.189+0000 7f5c361d0640 1 -- 172.21.15.28:0/1305688424 learned_addr learned my addr 172.21.15.28:0/1305688424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.190+0000 7f5c361d0640 1 -- 172.21.15.28:0/1305688424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c3006e9d0 msgr2=0x7f5c300ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.190+0000 7f5c361d0640 1 --2- 172.21.15.28:0/1305688424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c3006e9d0 0x7f5c300ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.190+0000 7f5c361d0640 1 -- 172.21.15.28:0/1305688424 >> v1:172.21.15.53:6789/0 conn(0x7f5c300ceec0 legacy=0x7f5c301647c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.190+0000 7f5c361d0640 1 -- 172.21.15.28:0/1305688424 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3006d940 con 0x7f5c300b9210 2024-04-25T19:13:33.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.190+0000 7f5c359cf640 1 --2- 172.21.15.28:0/1305688424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c3006e9d0 0x7f5c300ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.191+0000 7f5c361d0640 1 --2- 172.21.15.28:0/1305688424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300b9210 0x7f5c3006e490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009850 tx=0x7f5c2402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=419743c8dd6a7b55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.191+0000 7f5c351ce640 1 -- 172.21.15.28:0/1305688424 <== mon.1 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24004030 con 0x7f5c300b9210 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.191+0000 7f5c351ce640 1 -- 172.21.15.28:0/1305688424 <== mon.1 v2:172.21.15.53:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c240041d0 con 0x7f5c300b9210 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.191+0000 7f5c351ce640 1 -- 172.21.15.28:0/1305688424 <== mon.1 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24003610 con 0x7f5c300b9210 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.191+0000 7f5c38434640 1 -- 172.21.15.28:0/1305688424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300b9210 msgr2=0x7f5c3006e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.191+0000 7f5c38434640 1 --2- 172.21.15.28:0/1305688424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300b9210 0x7f5c3006e490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009850 tx=0x7f5c2402a1a0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.192+0000 7f5c38434640 1 -- 172.21.15.28:0/1305688424 shutdown_connections 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.192+0000 7f5c38434640 1 --2- 172.21.15.28:0/1305688424 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c3006e9d0 0x7f5c300ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.192+0000 7f5c38434640 1 --2- 172.21.15.28:0/1305688424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300b9210 0x7f5c3006e490 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.192+0000 7f5c38434640 1 -- 172.21.15.28:0/1305688424 >> 172.21.15.28:0/1305688424 conn(0x7f5c300d03a0 msgr2=0x7f5c300d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.192+0000 7f5c38434640 1 -- 172.21.15.28:0/1305688424 shutdown_connections 2024-04-25T19:13:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.192+0000 7f5c38434640 1 -- 172.21.15.28:0/1305688424 wait complete. 2024-04-25T19:13:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.193+0000 7f5c38434640 1 Processor -- start 2024-04-25T19:13:33.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.193+0000 7f5c38434640 1 -- start start 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c38434640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3006e9d0 0x7f5c3015dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c38434640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c300b9210 0x7f5c30152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c38434640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300ceec0 0x7f5c300b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c38434640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c300b8e60 con 0x7f5c3006e9d0 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c38434640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5c300b8ce0 con 0x7f5c300ceec0 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c38434640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5c30164ed0 con 0x7f5c300b9210 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3006e9d0 0x7f5c3015dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3006e9d0 0x7f5c3015dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55144/0 (socket says 172.21.15.28:55144) 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 -- 172.21.15.28:0/1700022180 learned_addr learned my addr 172.21.15.28:0/1700022180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:33.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c369d1640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300ceec0 0x7f5c300b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c359cf640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c300b9210 0x7f5c30152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 -- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c300b9210 msgr2=0x7f5c30152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c300b9210 0x7f5c30152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 -- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300ceec0 msgr2=0x7f5c300b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300ceec0 0x7f5c300b78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 -- 172.21.15.28:0/1700022180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3015e240 con 0x7f5c3006e9d0 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c359cf640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c300b9210 0x7f5c30152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c361d0640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3006e9d0 0x7f5c3015dd00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5c2402a1e0 tx=0x7f5c240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.196+0000 7f5c22ffd640 1 -- 172.21.15.28:0/1700022180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24041020 con 0x7f5c3006e9d0 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.197+0000 7f5c22ffd640 1 -- 172.21.15.28:0/1700022180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c24003c50 con 0x7f5c3006e9d0 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.197+0000 7f5c22ffd640 1 -- 172.21.15.28:0/1700022180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c24004030 con 0x7f5c3006e9d0 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.197+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c300c9dc0 con 0x7f5c3006e9d0 2024-04-25T19:13:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.197+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c300ca380 con 0x7f5c3006e9d0 2024-04-25T19:13:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.198+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c3006a100 con 0x7f5c3006e9d0 2024-04-25T19:13:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.198+0000 7f5c22ffd640 1 -- 172.21.15.28:0/1700022180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5c24044020 con 0x7f5c3006e9d0 2024-04-25T19:13:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.198+0000 7f5c22ffd640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c140744b0 0x7f5c140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.198+0000 7f5c22ffd640 1 -- 172.21.15.28:0/1700022180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 5869+0+0 (secure 0 0 0) 0x7f5c240b5d80 con 0x7f5c3006e9d0 2024-04-25T19:13:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.198+0000 7f5c359cf640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c140744b0 0x7f5c140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.198+0000 7f5c359cf640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c140744b0 0x7f5c140769c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5c300ceb80 tx=0x7f5c2c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.204+0000 7f5c22ffd640 1 -- 172.21.15.28:0/1700022180 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c2404b030 con 0x7f5c3006e9d0 2024-04-25T19:13:33.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:33.388+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5c300d17d0 con 0x7f5c3006e9d0 2024-04-25T19:13:34.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.048+0000 7f5c22ffd640 1 -- 172.21.15.28:0/1700022180 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v52) ==== 169+0+0 (secure 0 0 0) 0x7f5c2407feb0 con 0x7f5c3006e9d0 2024-04-25T19:13:34.050 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-04-25T19:13:34.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c140744b0 msgr2=0x7f5c140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c140744b0 0x7f5c140769c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5c300ceb80 tx=0x7f5c2c008040 comp rx=0 tx=0).stop 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3006e9d0 msgr2=0x7f5c3015dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3006e9d0 0x7f5c3015dd00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5c2402a1e0 tx=0x7f5c240046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 shutdown_connections 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c140744b0 0x7f5c140769c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c300ceec0 0x7f5c300b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c300b9210 0x7f5c30152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 --2- 172.21.15.28:0/1700022180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c3006e9d0 0x7f5c3015dd00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.050+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 >> 172.21.15.28:0/1700022180 conn(0x7f5c300d03a0 msgr2=0x7f5c30071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.051+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 shutdown_connections 2024-04-25T19:13:34.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.051+0000 7f5c38434640 1 -- 172.21.15.28:0/1700022180 wait complete. 2024-04-25T19:13:34.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T19:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.182+0000 7f282dd1f640 1 Processor -- start 2024-04-25T19:13:34.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.182+0000 7f282dd1f640 1 -- start start 2024-04-25T19:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.183+0000 7f282dd1f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 0x7f28280d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.183+0000 7f282dd1f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28280b79f0 0x7f28280b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f282dd1f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2828160720 con 0x7f282808cce0 2024-04-25T19:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f282dd1f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f28280b8340 con 0x7f28280b79f0 2024-04-25T19:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f282dd1f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f28280d5640 con 0x7f2828160310 2024-04-25T19:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f2826ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28280b79f0 0x7f28280b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f2827fff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f282808cce0 0x7f282808d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50854/0 (socket says 172.21.15.28:50854) 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f2826ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28280b79f0 0x7f28280b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55154/0 (socket says 172.21.15.28:55154) 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f2827fff640 1 -- 172.21.15.28:0/2291572534 learned_addr learned my addr 172.21.15.28:0/2291572534 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f28277fe640 1 --2- 172.21.15.28:0/2291572534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 0x7f28280d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f2826ffd640 1 -- 172.21.15.28:0/2291572534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 msgr2=0x7f28280d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f2826ffd640 1 --2- 172.21.15.28:0/2291572534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 0x7f28280d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f2826ffd640 1 -- 172.21.15.28:0/2291572534 >> v1:172.21.15.28:6789/0 conn(0x7f282808cce0 legacy=0x7f282808d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f28267fc640 1 -- 172.21.15.28:0/2291572534 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1878719609 0 0) 0x7f2828160720 con 0x7f282808cce0 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f2826ffd640 1 -- 172.21.15.28:0/2291572534 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28280d0f40 con 0x7f28280b79f0 2024-04-25T19:13:34.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.184+0000 7f28277fe640 1 --2- 172.21.15.28:0/2291572534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 0x7f28280d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.185+0000 7f2826ffd640 1 --2- 172.21.15.28:0/2291572534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28280b79f0 0x7f28280b7e00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f281c0097a0 tx=0x7f281c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758aa114ea820fe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.185+0000 7f28267fc640 1 -- 172.21.15.28:0/2291572534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c037070 con 0x7f28280b79f0 2024-04-25T19:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.185+0000 7f28267fc640 1 -- 172.21.15.28:0/2291572534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f281c032440 con 0x7f28280b79f0 2024-04-25T19:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.185+0000 7f28267fc640 1 -- 172.21.15.28:0/2291572534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281c032810 con 0x7f28280b79f0 2024-04-25T19:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.186+0000 7f282dd1f640 1 -- 172.21.15.28:0/2291572534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28280b79f0 msgr2=0x7f28280b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.186+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2291572534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28280b79f0 0x7f28280b7e00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f281c0097a0 tx=0x7f281c02a180 comp rx=0 tx=0).stop 2024-04-25T19:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.186+0000 7f282dd1f640 1 -- 172.21.15.28:0/2291572534 shutdown_connections 2024-04-25T19:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.186+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2291572534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28280b79f0 0x7f28280b7e00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.186+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2291572534 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 0x7f28280d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.186+0000 7f282dd1f640 1 -- 172.21.15.28:0/2291572534 >> 172.21.15.28:0/2291572534 conn(0x7f28280d2a50 msgr2=0x7f28280d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.186+0000 7f282dd1f640 1 -- 172.21.15.28:0/2291572534 shutdown_connections 2024-04-25T19:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.186+0000 7f282dd1f640 1 -- 172.21.15.28:0/2291572534 wait complete. 2024-04-25T19:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.188+0000 7f282dd1f640 1 Processor -- start 2024-04-25T19:13:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.188+0000 7f282dd1f640 1 -- start start 2024-04-25T19:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f282dd1f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282808cce0 0x7f28280c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f282dd1f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28280b79f0 0x7f282814b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f282dd1f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 0x7f282814be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f282dd1f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f28280b9400 con 0x7f282808cce0 2024-04-25T19:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f282dd1f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f28280d1500 con 0x7f28280b79f0 2024-04-25T19:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f282dd1f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f28280d0bf0 con 0x7f2828160310 2024-04-25T19:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f28277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282808cce0 0x7f28280c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f28277fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282808cce0 0x7f28280c8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55160/0 (socket says 172.21.15.28:55160) 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f28277fe640 1 -- 172.21.15.28:0/2808659990 learned_addr learned my addr 172.21.15.28:0/2808659990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f28277fe640 1 -- 172.21.15.28:0/2808659990 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 msgr2=0x7f282814be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f28277fe640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 0x7f282814be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f28277fe640 1 -- 172.21.15.28:0/2808659990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28280b79f0 msgr2=0x7f282814b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f28277fe640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28280b79f0 0x7f282814b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.189+0000 7f28277fe640 1 -- 172.21.15.28:0/2808659990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28280c93f0 con 0x7f282808cce0 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.190+0000 7f28277fe640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282808cce0 0x7f28280c8eb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f281800d800 tx=0x7f281800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.190+0000 7f280ffff640 1 -- 172.21.15.28:0/2808659990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2818004030 con 0x7f282808cce0 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.190+0000 7f280ffff640 1 -- 172.21.15.28:0/2808659990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28180041d0 con 0x7f282808cce0 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.190+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28280cdba0 con 0x7f282808cce0 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.190+0000 7f280ffff640 1 -- 172.21.15.28:0/2808659990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28180096a0 con 0x7f282808cce0 2024-04-25T19:13:34.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.190+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f282814b140 con 0x7f282808cce0 2024-04-25T19:13:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.191+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f282815ee40 con 0x7f282808cce0 2024-04-25T19:13:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.192+0000 7f280ffff640 1 -- 172.21.15.28:0/2808659990 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2818009840 con 0x7f282808cce0 2024-04-25T19:13:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.192+0000 7f280ffff640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f28140744b0 0x7f28140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.192+0000 7f280ffff640 1 -- 172.21.15.28:0/2808659990 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f28180953e0 con 0x7f282808cce0 2024-04-25T19:13:34.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.192+0000 7f2826ffd640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f28140744b0 0x7f28140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.193+0000 7f2826ffd640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f28140744b0 0x7f28140769c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f281c034000 tx=0x7f281c02c000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.197+0000 7f280ffff640 1 -- 172.21.15.28:0/2808659990 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f281805fc90 con 0x7f282808cce0 2024-04-25T19:13:34.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.385+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f282814b520 con 0x7f282808cce0 2024-04-25T19:13:34.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.389+0000 7f280ffff640 1 -- 172.21.15.28:0/2808659990 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v12) ==== 104+0+0 (secure 0 0 0) 0x7f281805f590 con 0x7f282808cce0 2024-04-25T19:13:34.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f28140744b0 msgr2=0x7f28140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f28140744b0 0x7f28140769c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f281c034000 tx=0x7f281c02c000 comp rx=0 tx=0).stop 2024-04-25T19:13:34.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282808cce0 msgr2=0x7f28280c8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282808cce0 0x7f28280c8eb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f281800d800 tx=0x7f281800dcc0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 shutdown_connections 2024-04-25T19:13:34.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f28140744b0 0x7f28140769c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2828160310 0x7f282814be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28280b79f0 0x7f282814b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 --2- 172.21.15.28:0/2808659990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f282808cce0 0x7f28280c8eb0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 >> 172.21.15.28:0/2808659990 conn(0x7f28280d2a50 msgr2=0x7f28280b9820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:34.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 shutdown_connections 2024-04-25T19:13:34.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.391+0000 7f282dd1f640 1 -- 172.21.15.28:0/2808659990 wait complete. 2024-04-25T19:13:34.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-04-25T19:13:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.519+0000 7fde2d106640 1 Processor -- start 2024-04-25T19:13:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.519+0000 7fde2d106640 1 -- start start 2024-04-25T19:13:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.520+0000 7fde2d106640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde28160310 0x7fde280d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.520+0000 7fde2d106640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde280b9920 0x7fde280b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.520+0000 7fde2d106640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde28160720 con 0x7fde280b79f0 2024-04-25T19:13:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.520+0000 7fde2d106640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fde280d5640 con 0x7fde28160310 2024-04-25T19:13:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.520+0000 7fde2d106640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fde280d0f40 con 0x7fde280b9920 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde26d9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde28160310 0x7fde280d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde2759e640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde280b9920 0x7fde280b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde26d9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde28160310 0x7fde280d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:37082/0 (socket says 172.21.15.28:37082) 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde26d9d640 1 -- 172.21.15.28:0/2956907119 learned_addr learned my addr 172.21.15.28:0/2956907119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde2759e640 1 -- 172.21.15.28:0/2956907119 >> v2:172.21.15.53:3300/0 conn(0x7fde28160310 msgr2=0x7fde280d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde2759e640 1 --2- 172.21.15.28:0/2956907119 >> v2:172.21.15.53:3300/0 conn(0x7fde28160310 0x7fde280d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde2759e640 1 -- 172.21.15.28:0/2956907119 >> v1:172.21.15.53:6789/0 conn(0x7fde280b79f0 legacy=0x7fde280b7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde2659c640 1 -- 172.21.15.28:0/2956907119 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096669402 0 0) 0x7fde28160720 con 0x7fde280b79f0 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde2759e640 1 -- 172.21.15.28:0/2956907119 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde280d1220 con 0x7fde280b9920 2024-04-25T19:13:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.521+0000 7fde26d9d640 1 --2- 172.21.15.28:0/2956907119 >> v2:172.21.15.53:3300/0 conn(0x7fde28160310 0x7fde280d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.522+0000 7fde2759e640 1 --2- 172.21.15.28:0/2956907119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde280b9920 0x7fde280b9d30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fde20009d90 tx=0x7fde2002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92546ed0811f1b69 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.522+0000 7fde2659c640 1 -- 172.21.15.28:0/2956907119 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde20007740 con 0x7fde280b9920 2024-04-25T19:13:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.522+0000 7fde2659c640 1 -- 172.21.15.28:0/2956907119 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde200078e0 con 0x7fde280b9920 2024-04-25T19:13:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.522+0000 7fde2659c640 1 -- 172.21.15.28:0/2956907119 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde20007cb0 con 0x7fde280b9920 2024-04-25T19:13:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.523+0000 7fde2d106640 1 -- 172.21.15.28:0/2956907119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde280b9920 msgr2=0x7fde280b9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.523+0000 7fde2d106640 1 --2- 172.21.15.28:0/2956907119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde280b9920 0x7fde280b9d30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fde20009d90 tx=0x7fde2002a660 comp rx=0 tx=0).stop 2024-04-25T19:13:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.523+0000 7fde2d106640 1 -- 172.21.15.28:0/2956907119 shutdown_connections 2024-04-25T19:13:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.523+0000 7fde2d106640 1 --2- 172.21.15.28:0/2956907119 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde280b9920 0x7fde280b9d30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.523+0000 7fde2d106640 1 --2- 172.21.15.28:0/2956907119 >> v2:172.21.15.53:3300/0 conn(0x7fde28160310 0x7fde280d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.523+0000 7fde2d106640 1 -- 172.21.15.28:0/2956907119 >> 172.21.15.28:0/2956907119 conn(0x7fde280d2a50 msgr2=0x7fde280d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.523+0000 7fde2d106640 1 -- 172.21.15.28:0/2956907119 shutdown_connections 2024-04-25T19:13:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.523+0000 7fde2d106640 1 -- 172.21.15.28:0/2956907119 wait complete. 2024-04-25T19:13:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.525+0000 7fde2d106640 1 Processor -- start 2024-04-25T19:13:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.525+0000 7fde2d106640 1 -- start start 2024-04-25T19:13:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.526+0000 7fde2d106640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde280b79f0 0x7fde2815c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.526+0000 7fde2d106640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde280b9920 0x7fde2814a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.526+0000 7fde2d106640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde28160310 0x7fde281f9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.526+0000 7fde2d106640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde280d0bf0 con 0x7fde280b79f0 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.526+0000 7fde2d106640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fde280b9400 con 0x7fde280b9920 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.526+0000 7fde2d106640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fde280cc950 con 0x7fde28160310 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde280b79f0 0x7fde2815c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde280b79f0 0x7fde2815c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55172/0 (socket says 172.21.15.28:55172) 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 -- 172.21.15.28:0/3906596010 learned_addr learned my addr 172.21.15.28:0/3906596010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde2759e640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde28160310 0x7fde281f9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde1e59c640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde280b9920 0x7fde2814a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 -- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde28160310 msgr2=0x7fde281f9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde28160310 0x7fde281f9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 -- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde280b9920 msgr2=0x7fde2814a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde280b9920 0x7fde2814a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 -- 172.21.15.28:0/3906596010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde2814add0 con 0x7fde280b79f0 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde2759e640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde28160310 0x7fde281f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde26d9d640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde280b79f0 0x7fde2815c710 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fde1800e9e0 tx=0x7fde1800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde1ffff640 1 -- 172.21.15.28:0/3906596010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde18017070 con 0x7fde280b79f0 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde280ca340 con 0x7fde280b79f0 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde1ffff640 1 -- 172.21.15.28:0/3906596010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde1800c9d0 con 0x7fde280b79f0 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde1ffff640 1 -- 172.21.15.28:0/3906596010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde1800cd80 con 0x7fde280b79f0 2024-04-25T19:13:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.527+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde280ca900 con 0x7fde280b79f0 2024-04-25T19:13:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.528+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde28050740 con 0x7fde280b79f0 2024-04-25T19:13:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.529+0000 7fde1ffff640 1 -- 172.21.15.28:0/3906596010 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fde1801a070 con 0x7fde280b79f0 2024-04-25T19:13:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.530+0000 7fde1ffff640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fde080744e0 0x7fde080769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.530+0000 7fde1ffff640 1 -- 172.21.15.28:0/3906596010 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7fde18095340 con 0x7fde280b79f0 2024-04-25T19:13:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.530+0000 7fde1e59c640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fde080744e0 0x7fde080769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.530+0000 7fde1e59c640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fde080744e0 0x7fde080769f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fde10000f60 tx=0x7fde10009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.534+0000 7fde1ffff640 1 -- 172.21.15.28:0/3906596010 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde1805f510 con 0x7fde280b79f0 2024-04-25T19:13:34.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.723+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) -- 0x7fde280d5cd0 con 0x7fde280b79f0 2024-04-25T19:13:34.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.726+0000 7fde1ffff640 1 -- 172.21.15.28:0/3906596010 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v13) ==== 92+0+0 (secure 0 0 0) 0x7fde18063810 con 0x7fde280b79f0 2024-04-25T19:13:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fde080744e0 msgr2=0x7fde080769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fde080744e0 0x7fde080769f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fde10000f60 tx=0x7fde10009040 comp rx=0 tx=0).stop 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde280b79f0 msgr2=0x7fde2815c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde280b79f0 0x7fde2815c710 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fde1800e9e0 tx=0x7fde1800eea0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 shutdown_connections 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fde080744e0 0x7fde080769f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fde28160310 0x7fde281f9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde280b9920 0x7fde2814a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 --2- 172.21.15.28:0/3906596010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde280b79f0 0x7fde2815c710 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.728+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 >> 172.21.15.28:0/3906596010 conn(0x7fde280d2a50 msgr2=0x7fde280d64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.729+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 shutdown_connections 2024-04-25T19:13:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.729+0000 7fde2d106640 1 -- 172.21.15.28:0/3906596010 wait complete. 2024-04-25T19:13:34.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple' 2024-04-25T19:13:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.852+0000 7f31de15f640 1 Processor -- start 2024-04-25T19:13:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.852+0000 7f31de15f640 1 -- start start 2024-04-25T19:13:34.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31de15f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d80cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31de15f640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31d806d710 con 0x7f31d809d010 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31de15f640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31d808ce70 con 0x7f31d80cef50 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31de15f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f31d808cc20 con 0x7f31d809cb60 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d77fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f31d809d010 0x7f31d806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50860/0 (socket says 172.21.15.28:50860) 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d77fe640 1 -- 172.21.15.28:0/3698846332 learned_addr learned my addr 172.21.15.28:0/3698846332 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d6ffd640 1 --2- 172.21.15.28:0/3698846332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d80cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d67fc640 1 -- 172.21.15.28:0/3698846332 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4157117183 0 0) 0x7f31d806d710 con 0x7f31d809d010 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d67fc640 1 -- 172.21.15.28:0/3698846332 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31c0003520 con 0x7f31d809d010 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d67fc640 1 -- 172.21.15.28:0/3698846332 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1147864732 0 0) 0x7f31d808ce70 con 0x7f31d80cef50 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d67fc640 1 -- 172.21.15.28:0/3698846332 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31d806d710 con 0x7f31d80cef50 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d67fc640 1 -- 172.21.15.28:0/3698846332 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2052863648 0 0) 0x7f31c0003520 con 0x7f31d809d010 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d6ffd640 1 -- 172.21.15.28:0/3698846332 >> v1:172.21.15.53:6789/0 conn(0x7f31d80cef50 legacy=0x7f31d80d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d6ffd640 1 -- 172.21.15.28:0/3698846332 >> v1:172.21.15.28:6789/0 conn(0x7f31d809d010 legacy=0x7f31d806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.854+0000 7f31d6ffd640 1 -- 172.21.15.28:0/3698846332 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d80b92d0 con 0x7f31d809cb60 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.855+0000 7f31d6ffd640 1 --2- 172.21.15.28:0/3698846332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d80cea10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0097d0 tx=0x7f31cc02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3747102ed7d37696 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.855+0000 7f31d67fc640 1 -- 172.21.15.28:0/3698846332 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc037070 con 0x7f31d809cb60 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.855+0000 7f31d67fc640 1 -- 172.21.15.28:0/3698846332 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc02ac30 con 0x7f31d809cb60 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.855+0000 7f31d67fc640 1 -- 172.21.15.28:0/3698846332 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8001810 con 0x7f31d809cb60 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.856+0000 7f31de15f640 1 -- 172.21.15.28:0/3698846332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 msgr2=0x7f31d80cea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.856+0000 7f31de15f640 1 --2- 172.21.15.28:0/3698846332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d80cea10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f31cc0097d0 tx=0x7f31cc02a1a0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.856+0000 7f31de15f640 1 -- 172.21.15.28:0/3698846332 shutdown_connections 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.856+0000 7f31de15f640 1 --2- 172.21.15.28:0/3698846332 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d80cea10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.856+0000 7f31de15f640 1 -- 172.21.15.28:0/3698846332 >> 172.21.15.28:0/3698846332 conn(0x7f31d80d0430 msgr2=0x7f31d80d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.856+0000 7f31de15f640 1 -- 172.21.15.28:0/3698846332 shutdown_connections 2024-04-25T19:13:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.856+0000 7f31de15f640 1 -- 172.21.15.28:0/3698846332 wait complete. 2024-04-25T19:13:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.858+0000 7f31de15f640 1 Processor -- start 2024-04-25T19:13:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.858+0000 7f31de15f640 1 -- start start 2024-04-25T19:13:34.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.859+0000 7f31de15f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d81d8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.859+0000 7f31de15f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d809d010 0x7f31d81d8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31de15f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31d80cef50 0x7f31d80a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31de15f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f31d806ea20 con 0x7f31d809d010 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31de15f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f31d806e5e0 con 0x7f31d80cef50 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31de15f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f31d806eba0 con 0x7f31d809cb60 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d809d010 0x7f31d81d8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d809d010 0x7f31d81d8680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55176/0 (socket says 172.21.15.28:55176) 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 -- 172.21.15.28:0/850262365 learned_addr learned my addr 172.21.15.28:0/850262365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d77fe640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d81d8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d7fff640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31d80cef50 0x7f31d80a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 -- 172.21.15.28:0/850262365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 msgr2=0x7f31d81d8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d81d8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 -- 172.21.15.28:0/850262365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31d80cef50 msgr2=0x7f31d80a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31d80cef50 0x7f31d80a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 -- 172.21.15.28:0/850262365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d81d8bc0 con 0x7f31d809d010 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31d6ffd640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d809d010 0x7f31d81d8680 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f31cc035040 tx=0x7f31cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31bffff640 1 -- 172.21.15.28:0/850262365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc037050 con 0x7f31d809d010 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31bffff640 1 -- 172.21.15.28:0/850262365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc043070 con 0x7f31d809d010 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d80cbb50 con 0x7f31d809d010 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.860+0000 7f31bffff640 1 -- 172.21.15.28:0/850262365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc006e10 con 0x7f31d809d010 2024-04-25T19:13:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.861+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d80cbe30 con 0x7f31d809d010 2024-04-25T19:13:34.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.861+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d8050740 con 0x7f31d809d010 2024-04-25T19:13:34.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.862+0000 7f31bffff640 1 -- 172.21.15.28:0/850262365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f31cc03ca10 con 0x7f31d809d010 2024-04-25T19:13:34.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.863+0000 7f31bffff640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31c40744b0 0x7f31c40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:34.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.863+0000 7f31d77fe640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31c40744b0 0x7f31c40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:34.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.863+0000 7f31bffff640 1 -- 172.21.15.28:0/850262365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f31cc0b5a60 con 0x7f31d809d010 2024-04-25T19:13:34.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.863+0000 7f31d77fe640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31c40744b0 0x7f31c40769c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f31c8001380 tx=0x7f31c8001cf0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:34.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:34.868+0000 7f31bffff640 1 -- 172.21.15.28:0/850262365 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31cc07fcc0 con 0x7f31d809d010 2024-04-25T19:13:35.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:35.129+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]} v 0) -- 0x7f31d8076100 con 0x7f31d809d010 2024-04-25T19:13:36.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.059+0000 7f31bffff640 1 -- 172.21.15.28:0/850262365 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_recovery_pool.TestRecoveryPool.test_rebuild_simple"]}]=0 v143) ==== 147+0+0 (secure 0 0 0) 0x7f31cc083fc0 con 0x7f31d809d010 2024-04-25T19:13:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31c40744b0 msgr2=0x7f31c40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31c40744b0 0x7f31c40769c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f31c8001380 tx=0x7f31c8001cf0 comp rx=0 tx=0).stop 2024-04-25T19:13:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d809d010 msgr2=0x7f31d81d8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:36.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d809d010 0x7f31d81d8680 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f31cc035040 tx=0x7f31cc0046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:36.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 shutdown_connections 2024-04-25T19:13:36.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f31c40744b0 0x7f31c40769c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:36.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31d80cef50 0x7f31d80a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:36.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31d809d010 0x7f31d81d8680 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:36.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 --2- 172.21.15.28:0/850262365 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f31d809cb60 0x7f31d81d8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:36.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 >> 172.21.15.28:0/850262365 conn(0x7f31d80d0430 msgr2=0x7f31d806ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:36.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 shutdown_connections 2024-04-25T19:13:36.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:36.062+0000 7f31de15f640 1 -- 172.21.15.28:0/850262365 wait complete. 2024-04-25T19:13:36.078 INFO:tasks.cephfs_test_runner:test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) ... ok 2024-04-25T19:13:36.078 INFO:tasks.cephfs_test_runner: 2024-04-25T19:13:36.079 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-25T19:13:36.079 INFO:tasks.cephfs_test_runner:Ran 1 test in 79.538s 2024-04-25T19:13:36.079 INFO:tasks.cephfs_test_runner: 2024-04-25T19:13:36.079 INFO:tasks.cephfs_test_runner:OK 2024-04-25T19:13:36.079 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-25T19:13:36.089 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T19:13:36.098 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T19:13:36.098 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:13:36.098 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:36.116 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16641 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T19:13:36.116 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-25T19:13:36.117 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:36.174 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16641 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T19:13:36.174 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:13:36.175 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:36.194 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=889 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T19:13:36.194 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-04-25T19:13:36.194 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T19:13:36.251 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=889 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T19:13:36.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.252 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:13:36.270 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T19:13:36.270 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T19:13:36.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T19:13:36.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T19:13:36.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-25T19:13:36.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.508 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-25T19:13:36.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-25T19:13:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-04-25T19:13:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:13:36.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.596 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-25T19:13:36.615 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T19:13:36.615 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T19:13:36.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.615 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T19:13:36.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T19:13:36.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-04-25T19:13:36.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.848 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-04-25T19:13:36.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T19:13:36.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-04-25T19:13:36.933 INFO:teuthology.orchestra.run.smithi053.stderr:Cannot find device "ceph-brx" 2024-04-25T19:13:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:13:36.935 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T19:13:36.944 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T19:13:36.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:13:37.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.067+0000 7fac64b4d640 1 Processor -- start 2024-04-25T19:13:37.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.067+0000 7fac64b4d640 1 -- start start 2024-04-25T19:13:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.068+0000 7fac64b4d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac600b76d0 0x7fac600b4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.068+0000 7fac64b4d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac600c89a0 con 0x7fac6006c570 2024-04-25T19:13:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.068+0000 7fac64b4d640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac600c8c40 con 0x7fac600ce950 2024-04-25T19:13:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.068+0000 7fac64b4d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fac600c6fb0 con 0x7fac600b76d0 2024-04-25T19:13:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.068+0000 7fac55d9b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fac6006c570 0x7fac60161160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50864/0 (socket says 172.21.15.28:50864) 2024-04-25T19:13:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.068+0000 7fac55d9b640 1 -- 172.21.15.28:0/1865082483 learned_addr learned my addr 172.21.15.28:0/1865082483 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.068+0000 7fac5e59c640 1 --2- 172.21.15.28:0/1865082483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac600b76d0 0x7fac600b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.069+0000 7fac5dd9b640 1 -- 172.21.15.28:0/1865082483 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2852992992 0 0) 0x7fac600c89a0 con 0x7fac6006c570 2024-04-25T19:13:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.069+0000 7fac5dd9b640 1 -- 172.21.15.28:0/1865082483 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac4c003520 con 0x7fac6006c570 2024-04-25T19:13:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.069+0000 7fac5dd9b640 1 -- 172.21.15.28:0/1865082483 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2365080651 0 0) 0x7fac600c8c40 con 0x7fac600ce950 2024-04-25T19:13:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.069+0000 7fac5dd9b640 1 -- 172.21.15.28:0/1865082483 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac600c89a0 con 0x7fac600ce950 2024-04-25T19:13:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.069+0000 7fac5e59c640 1 -- 172.21.15.28:0/1865082483 >> v1:172.21.15.53:6789/0 conn(0x7fac600ce950 legacy=0x7fac600ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.069+0000 7fac5e59c640 1 -- 172.21.15.28:0/1865082483 >> v1:172.21.15.28:6789/0 conn(0x7fac6006c570 legacy=0x7fac60161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.069+0000 7fac5e59c640 1 -- 172.21.15.28:0/1865082483 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6006c980 con 0x7fac600b76d0 2024-04-25T19:13:37.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac5e59c640 1 --2- 172.21.15.28:0/1865082483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac600b76d0 0x7fac600b4750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac500097d0 tx=0x7fac5002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f376ba00219b693 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac5dd9b640 1 -- 172.21.15.28:0/1865082483 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50037070 con 0x7fac600b76d0 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac5dd9b640 1 -- 172.21.15.28:0/1865082483 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac50032440 con 0x7fac600b76d0 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac5dd9b640 1 -- 172.21.15.28:0/1865082483 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50032810 con 0x7fac600b76d0 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac64b4d640 1 -- 172.21.15.28:0/1865082483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac600b76d0 msgr2=0x7fac600b4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac64b4d640 1 --2- 172.21.15.28:0/1865082483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac600b76d0 0x7fac600b4750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fac500097d0 tx=0x7fac5002a210 comp rx=0 tx=0).stop 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac64b4d640 1 -- 172.21.15.28:0/1865082483 shutdown_connections 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac64b4d640 1 --2- 172.21.15.28:0/1865082483 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac600b76d0 0x7fac600b4750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.070+0000 7fac64b4d640 1 -- 172.21.15.28:0/1865082483 >> 172.21.15.28:0/1865082483 conn(0x7fac6009b540 msgr2=0x7fac600b6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.071+0000 7fac64b4d640 1 -- 172.21.15.28:0/1865082483 shutdown_connections 2024-04-25T19:13:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.071+0000 7fac64b4d640 1 -- 172.21.15.28:0/1865082483 wait complete. 2024-04-25T19:13:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.073+0000 7fac64b4d640 1 Processor -- start 2024-04-25T19:13:37.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.073+0000 7fac64b4d640 1 -- start start 2024-04-25T19:13:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac64b4d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac6006c570 0x7fac60155130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac64b4d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac600b76d0 0x7fac600c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac64b4d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac600ce950 0x7fac600b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac64b4d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fac600c7bc0 con 0x7fac600ce950 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac64b4d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fac600c7a40 con 0x7fac600b76d0 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac64b4d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fac600c7d40 con 0x7fac6006c570 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac5ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac600ce950 0x7fac600b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac5ed9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac600ce950 0x7fac600b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55188/0 (socket says 172.21.15.28:55188) 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac5ed9d640 1 -- 172.21.15.28:0/2476691721 learned_addr learned my addr 172.21.15.28:0/2476691721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac5e59c640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac6006c570 0x7fac60155130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.074+0000 7fac55d9b640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac600b76d0 0x7fac600c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac5ed9d640 1 -- 172.21.15.28:0/2476691721 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac6006c570 msgr2=0x7fac60155130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac5ed9d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac6006c570 0x7fac60155130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac5ed9d640 1 -- 172.21.15.28:0/2476691721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac600b76d0 msgr2=0x7fac600c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac5ed9d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac600b76d0 0x7fac600c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac5ed9d640 1 -- 172.21.15.28:0/2476691721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac600cb830 con 0x7fac600ce950 2024-04-25T19:13:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac5ed9d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac600ce950 0x7fac600b0e80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fac5800c6b0 tx=0x7fac5800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac577fe640 1 -- 172.21.15.28:0/2476691721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58003b70 con 0x7fac600ce950 2024-04-25T19:13:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac577fe640 1 -- 172.21.15.28:0/2476691721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac58003d10 con 0x7fac600ce950 2024-04-25T19:13:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac600cbb10 con 0x7fac600ce950 2024-04-25T19:13:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac577fe640 1 -- 172.21.15.28:0/2476691721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58005310 con 0x7fac600ce950 2024-04-25T19:13:37.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.075+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac600cc0d0 con 0x7fac600ce950 2024-04-25T19:13:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.076+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac60050740 con 0x7fac600ce950 2024-04-25T19:13:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.077+0000 7fac577fe640 1 -- 172.21.15.28:0/2476691721 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fac580054b0 con 0x7fac600ce950 2024-04-25T19:13:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.077+0000 7fac577fe640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fac400744b0 0x7fac400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.077+0000 7fac577fe640 1 -- 172.21.15.28:0/2476691721 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7fac58094f40 con 0x7fac600ce950 2024-04-25T19:13:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.077+0000 7fac5e59c640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fac400744b0 0x7fac400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.078+0000 7fac5e59c640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fac400744b0 0x7fac400769c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fac50034000 tx=0x7fac5002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.082+0000 7fac577fe640 1 -- 172.21.15.28:0/2476691721 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac5805f110 con 0x7fac600ce950 2024-04-25T19:13:37.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.265+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fac600746d0 con 0x7fac400744b0 2024-04-25T19:13:37.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.266+0000 7fac577fe640 1 -- 172.21.15.28:0/2476691721 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13469 (secure 0 0 0) 0x7fac600746d0 con 0x7fac400744b0 2024-04-25T19:13:37.268 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fac400744b0 msgr2=0x7fac400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fac400744b0 0x7fac400769c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fac50034000 tx=0x7fac5002e000 comp rx=0 tx=0).stop 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac600ce950 msgr2=0x7fac600b0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac600ce950 0x7fac600b0e80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fac5800c6b0 tx=0x7fac5800cb70 comp rx=0 tx=0).stop 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 shutdown_connections 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fac400744b0 0x7fac400769c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fac600ce950 0x7fac600b0e80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac600b76d0 0x7fac600c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 --2- 172.21.15.28:0/2476691721 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fac6006c570 0x7fac60155130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 >> 172.21.15.28:0/2476691721 conn(0x7fac6009b540 msgr2=0x7fac6015e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 shutdown_connections 2024-04-25T19:13:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.268+0000 7fac64b4d640 1 -- 172.21.15.28:0/2476691721 wait complete. 2024-04-25T19:13:37.280 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-04-25T19:13:36.864980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":255492,"kb_used_data":6024,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719228,"statfs":{"total":773094113280,"available":772832489472,"internally_reserved":0,"allocated":6168576,"data_stored":3452920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322154,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.060975"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-25T19:13:34.056255+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:13:34.056255+0000","last_peered":"2024-04-25T19:13:34.056255+0000","last_clean":"2024-04-25T19:13:34.056255+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:13:34.056255+0000","last_undegraded":"2024-04-25T19:13:34.056255+0000","last_fullsized":"2024-04-25T19:13:34.056255+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:13:37.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:13:37.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.391+0000 7efde6755640 1 Processor -- start 2024-04-25T19:13:37.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.391+0000 7efde6755640 1 -- start start 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.392+0000 7efde6755640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.392+0000 7efde6755640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efde006e410 0x7efde00cea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.392+0000 7efde6755640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efde006d710 con 0x7efde009cb60 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.392+0000 7efde6755640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7efde008cc20 con 0x7efde006e410 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.392+0000 7efde6755640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efde008ce70 con 0x7efde009d010 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efde4cf2640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efde009cb60 0x7efde00cef50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50872/0 (socket says 172.21.15.28:50872) 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efde4cf2640 1 -- 172.21.15.28:0/1369673722 learned_addr learned my addr 172.21.15.28:0/1369673722 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efddffff640 1 --2- 172.21.15.28:0/1369673722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efddf7fe640 1 --2- 172.21.15.28:0/1369673722 >> v2:172.21.15.53:3300/0 conn(0x7efde006e410 0x7efde00cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efddeffd640 1 -- 172.21.15.28:0/1369673722 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4034439340 0 0) 0x7efde006d710 con 0x7efde009cb60 2024-04-25T19:13:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efddeffd640 1 -- 172.21.15.28:0/1369673722 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efdcc003520 con 0x7efde009cb60 2024-04-25T19:13:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efddffff640 1 -- 172.21.15.28:0/1369673722 >> v2:172.21.15.53:3300/0 conn(0x7efde006e410 msgr2=0x7efde00cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efddffff640 1 --2- 172.21.15.28:0/1369673722 >> v2:172.21.15.53:3300/0 conn(0x7efde006e410 0x7efde00cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efddffff640 1 -- 172.21.15.28:0/1369673722 >> v1:172.21.15.28:6789/0 conn(0x7efde009cb60 legacy=0x7efde00cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.393+0000 7efddffff640 1 -- 172.21.15.28:0/1369673722 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde00b92d0 con 0x7efde009d010 2024-04-25T19:13:37.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.394+0000 7efddffff640 1 --2- 172.21.15.28:0/1369673722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde006ded0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009790 tx=0x7efdc802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd95905156183bb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.394+0000 7efddeffd640 1 -- 172.21.15.28:0/1369673722 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc8037070 con 0x7efde009d010 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.394+0000 7efddeffd640 1 -- 172.21.15.28:0/1369673722 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc8032440 con 0x7efde009d010 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.394+0000 7efddeffd640 1 -- 172.21.15.28:0/1369673722 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc80327f0 con 0x7efde009d010 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.395+0000 7efde6755640 1 -- 172.21.15.28:0/1369673722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 msgr2=0x7efde006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.395+0000 7efde6755640 1 --2- 172.21.15.28:0/1369673722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde006ded0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdc8009790 tx=0x7efdc802a180 comp rx=0 tx=0).stop 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.395+0000 7efde6755640 1 -- 172.21.15.28:0/1369673722 shutdown_connections 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.395+0000 7efde6755640 1 --2- 172.21.15.28:0/1369673722 >> v2:172.21.15.53:3300/0 conn(0x7efde006e410 0x7efde00cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.395+0000 7efde6755640 1 --2- 172.21.15.28:0/1369673722 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde006ded0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.395+0000 7efde6755640 1 -- 172.21.15.28:0/1369673722 >> 172.21.15.28:0/1369673722 conn(0x7efde00d0430 msgr2=0x7efde00d0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.395+0000 7efde6755640 1 -- 172.21.15.28:0/1369673722 shutdown_connections 2024-04-25T19:13:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.395+0000 7efde6755640 1 -- 172.21.15.28:0/1369673722 wait complete. 2024-04-25T19:13:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.397+0000 7efde6755640 1 Processor -- start 2024-04-25T19:13:37.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.397+0000 7efde6755640 1 -- start start 2024-04-25T19:13:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efde6755640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efde006e410 0x7efde01dbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efde6755640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efde009cb60 0x7efde00cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efde6755640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde01f9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efde6755640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efde006eba0 con 0x7efde006e410 2024-04-25T19:13:37.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efde6755640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7efde006ea20 con 0x7efde009cb60 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efde6755640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7efde00d1cb0 con 0x7efde009d010 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efddffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efde006e410 0x7efde01dbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efddffff640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efde006e410 0x7efde01dbc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55204/0 (socket says 172.21.15.28:55204) 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efddffff640 1 -- 172.21.15.28:0/3082510612 learned_addr learned my addr 172.21.15.28:0/3082510612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efde4cf2640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde01f9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.398+0000 7efddf7fe640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efde009cb60 0x7efde00cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddffff640 1 -- 172.21.15.28:0/3082510612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 msgr2=0x7efde01f9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddffff640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde01f9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddffff640 1 -- 172.21.15.28:0/3082510612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efde009cb60 msgr2=0x7efde00cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddffff640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efde009cb60 0x7efde00cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddffff640 1 -- 172.21.15.28:0/3082510612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde00cc8e0 con 0x7efde006e410 2024-04-25T19:13:37.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddffff640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efde006e410 0x7efde01dbc90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efdc802a1c0 tx=0x7efdc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddcff9640 1 -- 172.21.15.28:0/3082510612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc8037040 con 0x7efde006e410 2024-04-25T19:13:37.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddcff9640 1 -- 172.21.15.28:0/3082510612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdc8042070 con 0x7efde006e410 2024-04-25T19:13:37.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde00cbe10 con 0x7efde006e410 2024-04-25T19:13:37.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efddcff9640 1 -- 172.21.15.28:0/3082510612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc803b930 con 0x7efde006e410 2024-04-25T19:13:37.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.399+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde00cc600 con 0x7efde006e410 2024-04-25T19:13:37.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.400+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde0050740 con 0x7efde006e410 2024-04-25T19:13:37.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.401+0000 7efddcff9640 1 -- 172.21.15.28:0/3082510612 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efdc803bad0 con 0x7efde006e410 2024-04-25T19:13:37.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.401+0000 7efddcff9640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efdc00744b0 0x7efdc00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.402+0000 7efddcff9640 1 -- 172.21.15.28:0/3082510612 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7efdc80b5940 con 0x7efde006e410 2024-04-25T19:13:37.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.402+0000 7efddf7fe640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efdc00744b0 0x7efdc00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.402+0000 7efddf7fe640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efdc00744b0 0x7efdc00769c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efde0073580 tx=0x7efdd0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.406+0000 7efddcff9640 1 -- 172.21.15.28:0/3082510612 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efdc8080310 con 0x7efde006e410 2024-04-25T19:13:37.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.590+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efde00b7230 con 0x7efdc00744b0 2024-04-25T19:13:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.591+0000 7efddcff9640 1 -- 172.21.15.28:0/3082510612 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13469 (secure 0 0 0) 0x7efde00b7230 con 0x7efdc00744b0 2024-04-25T19:13:37.593 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:13:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efdc00744b0 msgr2=0x7efdc00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efdc00744b0 0x7efdc00769c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7efde0073580 tx=0x7efdd0008040 comp rx=0 tx=0).stop 2024-04-25T19:13:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efde006e410 msgr2=0x7efde01dbc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efde006e410 0x7efde01dbc90 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7efdc802a1c0 tx=0x7efdc80046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 shutdown_connections 2024-04-25T19:13:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7efdc00744b0 0x7efdc00769c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7efde009d010 0x7efde01f9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efde009cb60 0x7efde00cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 --2- 172.21.15.28:0/3082510612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efde006e410 0x7efde01dbc90 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 >> 172.21.15.28:0/3082510612 conn(0x7efde00d0430 msgr2=0x7efde00b7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 shutdown_connections 2024-04-25T19:13:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.593+0000 7efde6755640 1 -- 172.21.15.28:0/3082510612 wait complete. 2024-04-25T19:13:37.606 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-04-25T19:13:36.864980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":255492,"kb_used_data":6024,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719228,"statfs":{"total":773094113280,"available":772832489472,"internally_reserved":0,"allocated":6168576,"data_stored":3452920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322154,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.060975"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-25T19:13:34.056255+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:13:34.056255+0000","last_peered":"2024-04-25T19:13:34.056255+0000","last_clean":"2024-04-25T19:13:34.056255+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:13:34.056255+0000","last_undegraded":"2024-04-25T19:13:34.056255+0000","last_fullsized":"2024-04-25T19:13:34.056255+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:13:37.606 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T19:13:37.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:13:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.717+0000 7f2da2d9d640 1 Processor -- start 2024-04-25T19:13:37.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.717+0000 7f2da2d9d640 1 -- start start 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da2d9d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2d9c0d1b60 0x7f2d9c0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da2d9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c06dc50 0x7f2d9c0b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da2d9d640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d9c0d36f0 0x7f2d9c0d3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da2d9d640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f2d9c0b3200 con 0x7f2d9c0d36f0 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da2d9d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f2d9c06e060 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da2d9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9c0d4040 con 0x7f2d9c06dc50 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da259c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d9c0d36f0 0x7f2d9c0d3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da259c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d9c0d36f0 0x7f2d9c0d3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55214/0 (socket says 172.21.15.28:55214) 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da259c640 1 -- 172.21.15.28:0/3319233105 learned_addr learned my addr 172.21.15.28:0/3319233105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da1d9b640 1 --2- 172.21.15.28:0/3319233105 >> v2:172.21.15.53:3300/0 conn(0x7f2d9c0d1b60 0x7f2d9c0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.718+0000 7f2da159a640 1 --2- 172.21.15.28:0/3319233105 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c06dc50 0x7f2d9c0b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.719+0000 7f2da259c640 1 -- 172.21.15.28:0/3319233105 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c06dc50 msgr2=0x7f2d9c0b2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.719+0000 7f2da259c640 1 --2- 172.21.15.28:0/3319233105 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c06dc50 0x7f2d9c0b2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.719+0000 7f2da259c640 1 -- 172.21.15.28:0/3319233105 >> v2:172.21.15.53:3300/0 conn(0x7f2d9c0d1b60 msgr2=0x7f2d9c0b8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.719+0000 7f2da259c640 1 --2- 172.21.15.28:0/3319233105 >> v2:172.21.15.53:3300/0 conn(0x7f2d9c0d1b60 0x7f2d9c0b8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.719+0000 7f2da259c640 1 -- 172.21.15.28:0/3319233105 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c0b9330 con 0x7f2d9c0d36f0 2024-04-25T19:13:37.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.719+0000 7f2da1d9b640 1 --2- 172.21.15.28:0/3319233105 >> v2:172.21.15.53:3300/0 conn(0x7f2d9c0d1b60 0x7f2d9c0b8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.719+0000 7f2da259c640 1 --2- 172.21.15.28:0/3319233105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d36f0 0x7f2d9c0d3b00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0d4440 tx=0x7f2d9402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9248ac11793997a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da0d99640 1 -- 172.21.15.28:0/3319233105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94037070 con 0x7f2d9c0d36f0 2024-04-25T19:13:37.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da0d99640 1 -- 172.21.15.28:0/3319233105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9402ae00 con 0x7f2d9c0d36f0 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da0d99640 1 -- 172.21.15.28:0/3319233105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94032660 con 0x7f2d9c0d36f0 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 -- 172.21.15.28:0/3319233105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d36f0 msgr2=0x7f2d9c0d3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/3319233105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d36f0 0x7f2d9c0d3b00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c0d4440 tx=0x7f2d9402a0b0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 -- 172.21.15.28:0/3319233105 shutdown_connections 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/3319233105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d36f0 0x7f2d9c0d3b00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/3319233105 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c06dc50 0x7f2d9c0b2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/3319233105 >> v2:172.21.15.53:3300/0 conn(0x7f2d9c0d1b60 0x7f2d9c0b8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 -- 172.21.15.28:0/3319233105 >> 172.21.15.28:0/3319233105 conn(0x7f2d9c08ce10 msgr2=0x7f2d9c08d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 -- 172.21.15.28:0/3319233105 shutdown_connections 2024-04-25T19:13:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.720+0000 7f2da2d9d640 1 -- 172.21.15.28:0/3319233105 wait complete. 2024-04-25T19:13:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.722+0000 7f2da2d9d640 1 Processor -- start 2024-04-25T19:13:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.722+0000 7f2da2d9d640 1 -- start start 2024-04-25T19:13:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.723+0000 7f2da2d9d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9c06dc50 0x7f2d9c15eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.723+0000 7f2da2d9d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d1b60 0x7f2d9c162f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.723+0000 7f2da2d9d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c0d36f0 0x7f2d9c159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.723+0000 7f2da2d9d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1644c0 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.723+0000 7f2da2d9d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1641e0 con 0x7f2d9c06dc50 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.723+0000 7f2da2d9d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9c1647a0 con 0x7f2d9c0d36f0 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d1b60 0x7f2d9c162f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d1b60 0x7f2d9c162f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55222/0 (socket says 172.21.15.28:55222) 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 -- 172.21.15.28:0/976478638 learned_addr learned my addr 172.21.15.28:0/976478638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da1d9b640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9c06dc50 0x7f2d9c15eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da259c640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c0d36f0 0x7f2d9c159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 -- 172.21.15.28:0/976478638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c0d36f0 msgr2=0x7f2d9c159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c0d36f0 0x7f2d9c159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 -- 172.21.15.28:0/976478638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9c06dc50 msgr2=0x7f2d9c15eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9c06dc50 0x7f2d9c15eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 -- 172.21.15.28:0/976478638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c15f510 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da159a640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d1b60 0x7f2d9c162f30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000e840 tx=0x7f2d9000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2d9a7fc640 1 -- 172.21.15.28:0/976478638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d90017070 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2d9a7fc640 1 -- 172.21.15.28:0/976478638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9000c730 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9c161890 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2d9a7fc640 1 -- 172.21.15.28:0/976478638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000cb00 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.724+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d9c161e50 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.725+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d9c050740 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.726+0000 7f2d9a7fc640 1 -- 172.21.15.28:0/976478638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2d9001a070 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.727+0000 7f2d9a7fc640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2d880744b0 0x7f2d880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.727+0000 7f2d9a7fc640 1 -- 172.21.15.28:0/976478638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f2d900952b0 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.727+0000 7f2da1d9b640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2d880744b0 0x7f2d880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.727+0000 7f2da1d9b640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2d880744b0 0x7f2d880769c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c003fd0 tx=0x7f2d8c0039f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.731+0000 7f2d9a7fc640 1 -- 172.21.15.28:0/976478638 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d9005fb60 con 0x7f2d9c0d1b60 2024-04-25T19:13:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.913+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2d9c0b36b0 con 0x7f2d880744b0 2024-04-25T19:13:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.914+0000 7f2d9a7fc640 1 -- 172.21.15.28:0/976478638 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13469 (secure 0 0 0) 0x7f2d9c0b36b0 con 0x7f2d880744b0 2024-04-25T19:13:37.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:13:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2d880744b0 msgr2=0x7f2d880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2d880744b0 0x7f2d880769c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c003fd0 tx=0x7f2d8c0039f0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d1b60 msgr2=0x7f2d9c162f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d1b60 0x7f2d9c162f30 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000e840 tx=0x7f2d9000ed00 comp rx=0 tx=0).stop 2024-04-25T19:13:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 shutdown_connections 2024-04-25T19:13:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2d880744b0 0x7f2d880769c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2d9c0d36f0 0x7f2d9c159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d9c0d1b60 0x7f2d9c162f30 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 --2- 172.21.15.28:0/976478638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9c06dc50 0x7f2d9c15eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 >> 172.21.15.28:0/976478638 conn(0x7f2d9c08ce10 msgr2=0x7f2d9c08d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 shutdown_connections 2024-04-25T19:13:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:37.916+0000 7f2da2d9d640 1 -- 172.21.15.28:0/976478638 wait complete. 2024-04-25T19:13:37.929 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-04-25T19:13:36.864980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":255492,"kb_used_data":6024,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719228,"statfs":{"total":773094113280,"available":772832489472,"internally_reserved":0,"allocated":6168576,"data_stored":3452920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6322154,"num_objects":-15,"num_object_clones":0,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":0,"num_read":-53,"num_read_kb":-39,"num_write":-50,"num_write_kb":-6193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.060975"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-25T19:13:34.056255+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:13:34.056255+0000","last_peered":"2024-04-25T19:13:34.056255+0000","last_clean":"2024-04-25T19:13:34.056255+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:13:34.056255+0000","last_undegraded":"2024-04-25T19:13:34.056255+0000","last_fullsized":"2024-04-25T19:13:34.056255+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:13:37.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T19:13:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.040+0000 7f4c54261640 1 Processor -- start 2024-04-25T19:13:38.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.040+0000 7f4c54261640 1 -- start start 2024-04-25T19:13:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.041+0000 7f4c54261640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c4c0b9190 0x7f4c4c09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.041+0000 7f4c54261640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4c4c0d1ad0 0x7f4c4c0b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.041+0000 7f4c54261640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c4c0d1030 con 0x7f4c4c0d16c0 2024-04-25T19:13:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.041+0000 7f4c54261640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f4c4c08cce0 con 0x7f4c4c0b9190 2024-04-25T19:13:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.041+0000 7f4c54261640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f4c4c0b73b0 con 0x7f4c4c0d1ad0 2024-04-25T19:13:38.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.041+0000 7f4c51ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c4c0b9190 0x7f4c4c09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:38.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.042+0000 7f4c51ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c4c0b9190 0x7f4c4c09cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55224/0 (socket says 172.21.15.28:55224) 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.042+0000 7f4c51ffd640 1 -- 172.21.15.28:0/1807415577 learned_addr learned my addr 172.21.15.28:0/1807415577 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.042+0000 7f4c527fe640 1 --2- 172.21.15.28:0/1807415577 >> v2:172.21.15.53:3300/0 conn(0x7f4c4c0d1ad0 0x7f4c4c0b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.042+0000 7f4c51ffd640 1 -- 172.21.15.28:0/1807415577 >> v2:172.21.15.53:3300/0 conn(0x7f4c4c0d1ad0 msgr2=0x7f4c4c0b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.042+0000 7f4c51ffd640 1 --2- 172.21.15.28:0/1807415577 >> v2:172.21.15.53:3300/0 conn(0x7f4c4c0d1ad0 0x7f4c4c0b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.042+0000 7f4c51ffd640 1 -- 172.21.15.28:0/1807415577 >> v1:172.21.15.53:6789/0 conn(0x7f4c4c0d16c0 legacy=0x7f4c4c0b6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.042+0000 7f4c51ffd640 1 -- 172.21.15.28:0/1807415577 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c0b8ab0 con 0x7f4c4c0b9190 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.043+0000 7f4c51ffd640 1 --2- 172.21.15.28:0/1807415577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c09cf20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0d3480 tx=0x7f4c4802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c266c3d2c85380 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.043+0000 7f4c50ffb640 1 -- 172.21.15.28:0/1807415577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48004030 con 0x7f4c4c0b9190 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.043+0000 7f4c50ffb640 1 -- 172.21.15.28:0/1807415577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c480041d0 con 0x7f4c4c0b9190 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.043+0000 7f4c50ffb640 1 -- 172.21.15.28:0/1807415577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48003610 con 0x7f4c4c0b9190 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.043+0000 7f4c54261640 1 -- 172.21.15.28:0/1807415577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 msgr2=0x7f4c4c09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.043+0000 7f4c54261640 1 --2- 172.21.15.28:0/1807415577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c09cf20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c0d3480 tx=0x7f4c4802a270 comp rx=0 tx=0).stop 2024-04-25T19:13:38.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.044+0000 7f4c54261640 1 -- 172.21.15.28:0/1807415577 shutdown_connections 2024-04-25T19:13:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.044+0000 7f4c54261640 1 --2- 172.21.15.28:0/1807415577 >> v2:172.21.15.53:3300/0 conn(0x7f4c4c0d1ad0 0x7f4c4c0b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.044+0000 7f4c54261640 1 --2- 172.21.15.28:0/1807415577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c09cf20 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.044+0000 7f4c54261640 1 -- 172.21.15.28:0/1807415577 >> 172.21.15.28:0/1807415577 conn(0x7f4c4c0d0390 msgr2=0x7f4c4c0d07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.044+0000 7f4c54261640 1 -- 172.21.15.28:0/1807415577 shutdown_connections 2024-04-25T19:13:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.044+0000 7f4c54261640 1 -- 172.21.15.28:0/1807415577 wait complete. 2024-04-25T19:13:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.045+0000 7f4c54261640 1 Processor -- start 2024-04-25T19:13:38.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.045+0000 7f4c54261640 1 -- start start 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c54261640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c1618f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c54261640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c4c0d16c0 0x7f4c4c14ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c54261640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c4c0d1ad0 0x7f4c4c156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c54261640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c0ce7e0 con 0x7f4c4c0b9190 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c54261640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f4c4c0b7530 con 0x7f4c4c0d1ad0 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c54261640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f4c4c0ce960 con 0x7f4c4c0d16c0 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c1618f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c1618f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55234/0 (socket says 172.21.15.28:55234) 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 -- 172.21.15.28:0/1382553339 learned_addr learned my addr 172.21.15.28:0/1382553339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c527fe640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c4c0d1ad0 0x7f4c4c156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c517fc640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c4c0d16c0 0x7f4c4c14ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 -- 172.21.15.28:0/1382553339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c4c0d16c0 msgr2=0x7f4c4c14ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c4c0d16c0 0x7f4c4c14ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 -- 172.21.15.28:0/1382553339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c4c0d1ad0 msgr2=0x7f4c4c156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c4c0d1ad0 0x7f4c4c156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 -- 172.21.15.28:0/1382553339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c4c1f7c50 con 0x7f4c4c0b9190 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.047+0000 7f4c51ffd640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c1618f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4c48034040 tx=0x7f4c480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.048+0000 7f4c427fc640 1 -- 172.21.15.28:0/1382553339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48041020 con 0x7f4c4c0b9190 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.048+0000 7f4c427fc640 1 -- 172.21.15.28:0/1382553339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c48003c50 con 0x7f4c4c0b9190 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.048+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c4c1f7f30 con 0x7f4c4c0b9190 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.048+0000 7f4c427fc640 1 -- 172.21.15.28:0/1382553339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c48004030 con 0x7f4c4c0b9190 2024-04-25T19:13:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.048+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c4c1f84f0 con 0x7f4c4c0b9190 2024-04-25T19:13:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.049+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c4c050740 con 0x7f4c4c0b9190 2024-04-25T19:13:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.049+0000 7f4c427fc640 1 -- 172.21.15.28:0/1382553339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4c48044020 con 0x7f4c4c0b9190 2024-04-25T19:13:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.050+0000 7f4c427fc640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4c300744b0 0x7f4c300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.050+0000 7f4c427fc640 1 -- 172.21.15.28:0/1382553339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f4c480b5b40 con 0x7f4c4c0b9190 2024-04-25T19:13:38.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.050+0000 7f4c517fc640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4c300744b0 0x7f4c300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.051+0000 7f4c517fc640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4c300744b0 0x7f4c300769c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c06ca80 tx=0x7f4c3c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:38.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.055+0000 7f4c427fc640 1 -- 172.21.15.28:0/1382553339 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c4804b030 con 0x7f4c4c0b9190 2024-04-25T19:13:38.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.242+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4c4c0cfaf0 con 0x7f4c4c0b9190 2024-04-25T19:13:38.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.243+0000 7f4c427fc640 1 -- 172.21.15.28:0/1382553339 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+13361 (secure 0 0 0) 0x7f4c4807fda0 con 0x7f4c4c0b9190 2024-04-25T19:13:38.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:38.246 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":52,"fsid":"5d8007d4-24a5-41be-aa4d-cd64fedac62a","created":"2024-04-25T19:11:04.557577+0000","modified":"2024-04-25T19:13:34.047681+0000","last_up_change":"2024-04-25T19:11:15.310184+0000","last_in_change":"2024-04-25T19:11:06.822340+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T19:11:16.852809+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f1ad1667-8dae-42ec-9e8e-d7b1e8a18dbd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6801","nonce":1137456103}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6803","nonce":1137456103}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6807","nonce":1137456103}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1137456103},{"type":"v1","addr":"172.21.15.28:6805","nonce":1137456103}]},"public_addr":"172.21.15.28:6801/1137456103","cluster_addr":"172.21.15.28:6803/1137456103","heartbeat_back_addr":"172.21.15.28:6807/1137456103","heartbeat_front_addr":"172.21.15.28:6805/1137456103","state":["exists","up"]},{"osd":1,"uuid":"19c91f41-b727-4e4a-a862-4a042b38ef05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6809","nonce":1898594658}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6811","nonce":1898594658}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6815","nonce":1898594658}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":1898594658},{"type":"v1","addr":"172.21.15.28:6813","nonce":1898594658}]},"public_addr":"172.21.15.28:6809/1898594658","cluster_addr":"172.21.15.28:6811/1898594658","heartbeat_back_addr":"172.21.15.28:6815/1898594658","heartbeat_front_addr":"172.21.15.28:6813/1898594658","state":["exists","up"]},{"osd":2,"uuid":"6f313a73-d3d6-48af-bc8c-6d9faabc7b7f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6817","nonce":3410596961}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6819","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6821","nonce":3410596961}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6827","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6829","nonce":3410596961}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6823","nonce":3410596961},{"type":"v1","addr":"172.21.15.28:6825","nonce":3410596961}]},"public_addr":"172.21.15.28:6817/3410596961","cluster_addr":"172.21.15.28:6821/3410596961","heartbeat_back_addr":"172.21.15.28:6829/3410596961","heartbeat_front_addr":"172.21.15.28:6825/3410596961","state":["exists","up"]},{"osd":3,"uuid":"420b3523-29ab-4043-bc1b-2457f61a5e04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6820","nonce":1928233777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6824","nonce":1928233777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6831","nonce":1928233777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":1928233777},{"type":"v1","addr":"172.21.15.28:6828","nonce":1928233777}]},"public_addr":"172.21.15.28:6820/1928233777","cluster_addr":"172.21.15.28:6824/1928233777","heartbeat_back_addr":"172.21.15.28:6831/1928233777","heartbeat_front_addr":"172.21.15.28:6828/1928233777","state":["exists","up"]},{"osd":4,"uuid":"fceab193-1664-4927-9f5e-b04ae6651271","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6825","nonce":2055146181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6827","nonce":2055146181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6831","nonce":2055146181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":2055146181},{"type":"v1","addr":"172.21.15.53:6829","nonce":2055146181}]},"public_addr":"172.21.15.53:6825/2055146181","cluster_addr":"172.21.15.53:6827/2055146181","heartbeat_back_addr":"172.21.15.53:6831/2055146181","heartbeat_front_addr":"172.21.15.53:6829/2055146181","state":["exists","up"]},{"osd":5,"uuid":"041516ce-9cb0-49e2-9993-5a3968f84727","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6801","nonce":2342659583}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6803","nonce":2342659583}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6807","nonce":2342659583}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2342659583},{"type":"v1","addr":"172.21.15.53:6805","nonce":2342659583}]},"public_addr":"172.21.15.53:6801/2342659583","cluster_addr":"172.21.15.53:6803/2342659583","heartbeat_back_addr":"172.21.15.53:6807/2342659583","heartbeat_front_addr":"172.21.15.53:6805/2342659583","state":["exists","up"]},{"osd":6,"uuid":"cb5e7877-ce29-42c3-9557-3e1f3cc2395f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6809","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6811","nonce":3653668189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6813","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6815","nonce":3653668189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6821","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6823","nonce":3653668189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6817","nonce":3653668189},{"type":"v1","addr":"172.21.15.53:6819","nonce":3653668189}]},"public_addr":"172.21.15.53:6811/3653668189","cluster_addr":"172.21.15.53:6815/3653668189","heartbeat_back_addr":"172.21.15.53:6823/3653668189","heartbeat_front_addr":"172.21.15.53:6819/3653668189","state":["exists","up"]},{"osd":7,"uuid":"07a614e4-04f7-4d78-a275-f4a9c3267e8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6810","nonce":79451145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6814","nonce":79451145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6822","nonce":79451145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":79451145},{"type":"v1","addr":"172.21.15.53:6818","nonce":79451145}]},"public_addr":"172.21.15.53:6810/79451145","cluster_addr":"172.21.15.53:6814/79451145","heartbeat_back_addr":"172.21.15.53:6822/79451145","heartbeat_front_addr":"172.21.15.53:6818/79451145","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.942527+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.931433+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.960972+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.062205+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.006376+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.989948+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:11.951770+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T19:11:12.003655+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6837/1842070529":"2024-04-26T19:13:30.801760+0000","172.21.15.28:6836/1842070529":"2024-04-26T19:13:30.801760+0000","172.21.15.53:6835/803470605":"2024-04-26T19:12:54.688887+0000","172.21.15.53:6833/803470605":"2024-04-26T19:12:54.688887+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T19:13:38.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4c300744b0 msgr2=0x7f4c300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4c300744b0 0x7f4c300769c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c06ca80 tx=0x7f4c3c009040 comp rx=0 tx=0).stop 2024-04-25T19:13:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 msgr2=0x7f4c4c1618f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c1618f0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4c48034040 tx=0x7f4c480046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 shutdown_connections 2024-04-25T19:13:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f4c300744b0 0x7f4c300769c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c4c0d1ad0 0x7f4c4c156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f4c4c0d16c0 0x7f4c4c14ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 --2- 172.21.15.28:0/1382553339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c4c0b9190 0x7f4c4c1618f0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 >> 172.21.15.28:0/1382553339 conn(0x7f4c4c0d0390 msgr2=0x7f4c4c0723a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 shutdown_connections 2024-04-25T19:13:38.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:38.245+0000 7f4c54261640 1 -- 172.21.15.28:0/1382553339 wait complete. 2024-04-25T19:13:39.259 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T19:13:39.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T19:13:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.377+0000 7f71f5a61640 1 Processor -- start 2024-04-25T19:13:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.377+0000 7f71f5a61640 1 -- start start 2024-04-25T19:13:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.378+0000 7f71f5a61640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71f00b61c0 0x7f71f006f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.378+0000 7f71f5a61640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f00afcb0 con 0x7f71f015de20 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.378+0000 7f71f5a61640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f00aed60 con 0x7f71f00ae880 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.378+0000 7f71f5a61640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f71f00af040 con 0x7f71f00b61c0 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.378+0000 7f71ef7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71f00b61c0 0x7f71f006f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.378+0000 7f71ef7fe640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71f00b61c0 0x7f71f006f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55244/0 (socket says 172.21.15.28:55244) 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.378+0000 7f71ef7fe640 1 -- 172.21.15.28:0/340509802 learned_addr learned my addr 172.21.15.28:0/340509802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.378+0000 7f71e67fc640 1 --1- 172.21.15.28:0/340509802 >> v1:172.21.15.28:6789/0 conn(0x7f71f015de20 0x7f71f00b5ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50882/0 (socket says 172.21.15.28:50882) 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.379+0000 7f71ef7fe640 1 -- 172.21.15.28:0/340509802 >> v1:172.21.15.53:6789/0 conn(0x7f71f00ae880 legacy=0x7f71f00cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.379+0000 7f71ef7fe640 1 -- 172.21.15.28:0/340509802 >> v1:172.21.15.28:6789/0 conn(0x7f71f015de20 legacy=0x7f71f00b5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.379+0000 7f71ef7fe640 1 -- 172.21.15.28:0/340509802 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f00af320 con 0x7f71f00b61c0 2024-04-25T19:13:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.379+0000 7f71ee7fc640 1 -- 172.21.15.28:0/340509802 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3510137194 0 0) 0x7f71f00afcb0 con 0x7f71f015de20 2024-04-25T19:13:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.379+0000 7f71ef7fe640 1 --2- 172.21.15.28:0/340509802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b61c0 0x7f71f006f4c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f71e800d150 tx=0x7f71e802c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d16d586381e5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71ee7fc640 1 -- 172.21.15.28:0/340509802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e8035020 con 0x7f71f00b61c0 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71ee7fc640 1 -- 172.21.15.28:0/340509802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e800b2d0 con 0x7f71f00b61c0 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71ee7fc640 1 -- 172.21.15.28:0/340509802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e800b480 con 0x7f71f00b61c0 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71f5a61640 1 -- 172.21.15.28:0/340509802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b61c0 msgr2=0x7f71f006f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71f5a61640 1 --2- 172.21.15.28:0/340509802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b61c0 0x7f71f006f4c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f71e800d150 tx=0x7f71e802c9c0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71f5a61640 1 -- 172.21.15.28:0/340509802 shutdown_connections 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71f5a61640 1 --2- 172.21.15.28:0/340509802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b61c0 0x7f71f006f4c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71f5a61640 1 -- 172.21.15.28:0/340509802 >> 172.21.15.28:0/340509802 conn(0x7f71f0073920 msgr2=0x7f71f0073d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71f5a61640 1 -- 172.21.15.28:0/340509802 shutdown_connections 2024-04-25T19:13:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.380+0000 7f71f5a61640 1 -- 172.21.15.28:0/340509802 wait complete. 2024-04-25T19:13:39.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.382+0000 7f71f5a61640 1 Processor -- start 2024-04-25T19:13:39.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.382+0000 7f71f5a61640 1 -- start start 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71f5a61640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00ae880 0x7f71f013bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71f5a61640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f015de20 0x7f71f0140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71f5a61640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71f00d0610 0x7f71f013cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71f5a61640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71f00714b0 con 0x7f71f00ae880 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71f5a61640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f71f015e230 con 0x7f71f015de20 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71f5a61640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f71f0071630 con 0x7f71f00d0610 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00ae880 0x7f71f013bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00ae880 0x7f71f013bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55260/0 (socket says 172.21.15.28:55260) 2024-04-25T19:13:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 -- 172.21.15.28:0/2875734926 learned_addr learned my addr 172.21.15.28:0/2875734926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71e67fc640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f015de20 0x7f71f0140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71ef7fe640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71f00d0610 0x7f71f013cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 -- 172.21.15.28:0/2875734926 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71f00d0610 msgr2=0x7f71f013cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f71f00d0610 0x7f71f013cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 -- 172.21.15.28:0/2875734926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f015de20 msgr2=0x7f71f0140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f015de20 0x7f71f0140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 -- 172.21.15.28:0/2875734926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f00cf2b0 con 0x7f71f00ae880 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71eeffd640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00ae880 0x7f71f013bae0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f71e000c400 tx=0x7f71e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0017070 con 0x7f71f00ae880 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e00052d0 con 0x7f71f00ae880 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.384+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f00cf570 con 0x7f71f00ae880 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.385+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0003830 con 0x7f71f00ae880 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.385+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f00cfb30 con 0x7f71f00ae880 2024-04-25T19:13:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.385+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f71f006a100 con 0x7f71f00ae880 2024-04-25T19:13:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.386+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f71e001a070 con 0x7f71f00ae880 2024-04-25T19:13:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71ecff9640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71d00744b0 0x7f71d00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f71e0094550 con 0x7f71f00ae880 2024-04-25T19:13:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71e67fc640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71d00744b0 0x7f71d00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71ecff9640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f71d007aa40 0x7f71d007ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71d007d5a0 con 0x7f71d007aa40 2024-04-25T19:13:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f71e009b050 con 0x7f71f00ae880 2024-04-25T19:13:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71ef7fe640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f71d007aa40 0x7f71d007ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71e67fc640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71d00744b0 0x7f71d00769c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f71f01432d0 tx=0x7f71d8007d20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.387+0000 7f71ef7fe640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f71d007aa40 0x7f71d007ced0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.388+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 <== osd.0 v2:172.21.15.28:6800/1137456103 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f71d007d5a0 con 0x7f71d007aa40 2024-04-25T19:13:39.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.401+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 --> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f71f0070280 con 0x7f71d007aa40 2024-04-25T19:13:39.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71ecff9640 1 -- 172.21.15.28:0/2875734926 <== osd.0 v2:172.21.15.28:6800/1137456103 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f71f0070280 con 0x7f71d007aa40 2024-04-25T19:13:39.403 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:39.404 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T19:13:39.404 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f71d007aa40 msgr2=0x7f71d007ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6800/1137456103,v1:172.21.15.28:6801/1137456103] conn(0x7f71d007aa40 0x7f71d007ced0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71d00744b0 msgr2=0x7f71d00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71d00744b0 0x7f71d00769c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f71f01432d0 tx=0x7f71d8007d20 comp rx=0 tx=0).stop 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00ae880 msgr2=0x7f71f013bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 --2- 172.21.15.28:0/2875734926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00ae880 0x7f71f013bae0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f71e000c400 tx=0x7f71e000c8c0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71ef7fe640 1 -- 172.21.15.28:0/2875734926 reap_dead start 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 shutdown_connections 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 >> 172.21.15.28:0/2875734926 conn(0x7f71f0073920 msgr2=0x7f71f0146000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 shutdown_connections 2024-04-25T19:13:39.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.402+0000 7f71f5a61640 1 -- 172.21.15.28:0/2875734926 wait complete. 2024-04-25T19:13:39.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T19:13:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.525+0000 7f5193c42640 1 Processor -- start 2024-04-25T19:13:39.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.526+0000 7f5193c42640 1 -- start start 2024-04-25T19:13:39.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f5193c42640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f518c15aaa0 0x7f518c0affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f5193c42640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 0x7f518c160990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f5193c42640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f518c06f5b0 con 0x7f518c0d0ee0 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f5193c42640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f518c0af290 con 0x7f518c15aaa0 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f5193c42640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f518c0d14a0 con 0x7f518c0b5020 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f51911dd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f518c0d0ee0 0x7f518c0b4910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50890/0 (socket says 172.21.15.28:50890) 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f51911dd640 1 -- 172.21.15.28:0/4199255814 learned_addr learned my addr 172.21.15.28:0/4199255814 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f51921df640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 0x7f518c160990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f51919de640 1 --2- 172.21.15.28:0/4199255814 >> v2:172.21.15.53:3300/0 conn(0x7f518c15aaa0 0x7f518c0affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.527+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 660013482 0 0) 0x7f518c06f5b0 con 0x7f518c0d0ee0 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5178003520 con 0x7f518c0d0ee0 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 118554982 0 0) 0x7f5178003520 con 0x7f518c0d0ee0 2024-04-25T19:13:39.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 msgr2=0x7f518c160990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 0x7f518c160990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 >> v2:172.21.15.53:3300/0 conn(0x7f518c15aaa0 msgr2=0x7f518c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 --2- 172.21.15.28:0/4199255814 >> v2:172.21.15.53:3300/0 conn(0x7f518c15aaa0 0x7f518c0affc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c0af4e0 con 0x7f518c0d0ee0 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f51880019f0 con 0x7f518c0d0ee0 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 >> v1:172.21.15.28:6789/0 conn(0x7f518c0d0ee0 legacy=0x7f518c0b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51780039b0 0x7f5178003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5178004800 0x7f5178004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 0x7f517800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f518c0af290 con 0x7f51780039b0 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f518c0d14a0 con 0x7f5178004800 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f51780052d0 con 0x7f518c0b5020 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51919de640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51780039b0 0x7f5178003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51911dd640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5178004800 0x7f5178004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51921df640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 0x7f517800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51919de640 1 -- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 msgr2=0x7f517800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51919de640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 0x7f517800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51919de640 1 -- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5178004800 msgr2=0x7f5178004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51919de640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5178004800 0x7f5178004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51919de640 1 -- 172.21.15.28:0/4199255814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5178003520 con 0x7f51780039b0 2024-04-25T19:13:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.528+0000 7f51921df640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c0b5020 0x7f517800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:39.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.529+0000 7f51919de640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51780039b0 0x7f5178003e20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5180009ac0 tx=0x7f5178005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45666bbd78eac84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.529+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180036070 con 0x7f51780039b0 2024-04-25T19:13:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.529+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5180031400 con 0x7f51780039b0 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.529+0000 7f51909dc640 1 -- 172.21.15.28:0/4199255814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5180009690 con 0x7f51780039b0 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.530+0000 7f5193c42640 1 -- 172.21.15.28:0/4199255814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51780039b0 msgr2=0x7f5178003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.530+0000 7f5193c42640 1 --2- 172.21.15.28:0/4199255814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51780039b0 0x7f5178003e20 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f5180009ac0 tx=0x7f5178005830 comp rx=0 tx=0).stop 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.530+0000 7f51921df640 1 -- 172.21.15.28:0/4199255814 reap_dead start 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.530+0000 7f5193c42640 1 -- 172.21.15.28:0/4199255814 shutdown_connections 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.530+0000 7f5193c42640 1 -- 172.21.15.28:0/4199255814 >> 172.21.15.28:0/4199255814 conn(0x7f518c0742c0 msgr2=0x7f518c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.530+0000 7f5193c42640 1 -- 172.21.15.28:0/4199255814 shutdown_connections 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.530+0000 7f5193c42640 1 -- 172.21.15.28:0/4199255814 wait complete. 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.532+0000 7f5193c42640 1 Processor -- start 2024-04-25T19:13:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.532+0000 7f5193c42640 1 -- start start 2024-04-25T19:13:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.533+0000 7f5193c42640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51780039b0 0x7f518c151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.533+0000 7f5193c42640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c156100 0x7f518c1483a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.533+0000 7f5193c42640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c153d60 0x7f518c154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.533+0000 7f5193c42640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f518c0b5430 con 0x7f518c153d60 2024-04-25T19:13:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.533+0000 7f5193c42640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f518c0afaa0 con 0x7f51780039b0 2024-04-25T19:13:39.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.533+0000 7f5193c42640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f518c0b55b0 con 0x7f518c156100 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.533+0000 7f51921df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c153d60 0x7f518c154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51921df640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c153d60 0x7f518c154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55290/0 (socket says 172.21.15.28:55290) 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51921df640 1 -- 172.21.15.28:0/473007655 learned_addr learned my addr 172.21.15.28:0/473007655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51911dd640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c156100 0x7f518c1483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51919de640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51780039b0 0x7f518c151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51921df640 1 -- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c156100 msgr2=0x7f518c1483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51921df640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c156100 0x7f518c1483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51921df640 1 -- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51780039b0 msgr2=0x7f518c151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51921df640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51780039b0 0x7f518c151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51921df640 1 -- 172.21.15.28:0/473007655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518c152640 con 0x7f518c153d60 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51911dd640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c156100 0x7f518c1483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51919de640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51780039b0 0x7f518c151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f51921df640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c153d60 0x7f518c154170 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f518c0732f0 tx=0x7f5184011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f517e7fc640 1 -- 172.21.15.28:0/473007655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5184019020 con 0x7f518c153d60 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f517e7fc640 1 -- 172.21.15.28:0/473007655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5184003610 con 0x7f518c153d60 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f518c152920 con 0x7f518c153d60 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f517e7fc640 1 -- 172.21.15.28:0/473007655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5184003a00 con 0x7f518c153d60 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.534+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f518c153840 con 0x7f518c153d60 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.535+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f518c1523b0 con 0x7f518c153d60 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.536+0000 7f517e7fc640 1 -- 172.21.15.28:0/473007655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f518401c020 con 0x7f518c153d60 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.536+0000 7f517e7fc640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f51700744e0 0x7f51700769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.537+0000 7f517e7fc640 1 -- 172.21.15.28:0/473007655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f5184095320 con 0x7f518c153d60 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.537+0000 7f51919de640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f51700744e0 0x7f51700769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.537+0000 7f51919de640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f51700744e0 0x7f51700769f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5180033000 tx=0x7f5180008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.541+0000 7f517e7fc640 1 -- 172.21.15.28:0/473007655 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5184023030 con 0x7f518c153d60 2024-04-25T19:13:39.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.725+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f518c152e40 con 0x7f51700744e0 2024-04-25T19:13:39.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.726+0000 7f517e7fc640 1 -- 172.21.15.28:0/473007655 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f518c152e40 con 0x7f51700744e0 2024-04-25T19:13:39.728 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T19:13:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f51700744e0 msgr2=0x7f51700769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f51700744e0 0x7f51700769f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5180033000 tx=0x7f5180008030 comp rx=0 tx=0).stop 2024-04-25T19:13:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c153d60 msgr2=0x7f518c154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c153d60 0x7f518c154170 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f518c0732f0 tx=0x7f5184011920 comp rx=0 tx=0).stop 2024-04-25T19:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 shutdown_connections 2024-04-25T19:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f51700744e0 0x7f51700769f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f518c153d60 0x7f518c154170 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f518c156100 0x7f518c1483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 --2- 172.21.15.28:0/473007655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51780039b0 0x7f518c151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 >> 172.21.15.28:0/473007655 conn(0x7f518c0742c0 msgr2=0x7f518c15b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 shutdown_connections 2024-04-25T19:13:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.728+0000 7f5193c42640 1 -- 172.21.15.28:0/473007655 wait complete. 2024-04-25T19:13:39.741 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T19:13:39.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T19:13:39.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.856+0000 7f6d7c5c0640 1 Processor -- start 2024-04-25T19:13:39.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.856+0000 7f6d7c5c0640 1 -- start start 2024-04-25T19:13:39.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d7c5c0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d740b8eb0 0x7f6d7409bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d7c5c0640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6d7408cbb0 0x7f6d740d1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d7c5c0640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d740cf800 con 0x7f6d7409c270 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d7c5c0640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f6d740b43b0 con 0x7f6d740b8eb0 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d7c5c0640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f6d7409cf20 con 0x7f6d7408cbb0 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d7a35c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d740b8eb0 0x7f6d7409bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d79b5b640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6d7409c270 0x7f6d740d1760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50904/0 (socket says 172.21.15.28:50904) 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d79b5b640 1 -- 172.21.15.28:0/3396342514 learned_addr learned my addr 172.21.15.28:0/3396342514 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.857+0000 7f6d7ab5d640 1 --2- 172.21.15.28:0/3396342514 >> v2:172.21.15.53:3300/0 conn(0x7f6d7408cbb0 0x7f6d740d1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.858+0000 7f6d7a35c640 1 -- 172.21.15.28:0/3396342514 >> v2:172.21.15.53:3300/0 conn(0x7f6d7408cbb0 msgr2=0x7f6d740d1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.858+0000 7f6d7a35c640 1 --2- 172.21.15.28:0/3396342514 >> v2:172.21.15.53:3300/0 conn(0x7f6d7408cbb0 0x7f6d740d1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.858+0000 7f6d7a35c640 1 -- 172.21.15.28:0/3396342514 >> v1:172.21.15.28:6789/0 conn(0x7f6d7409c270 legacy=0x7f6d740d1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.858+0000 7f6d7a35c640 1 -- 172.21.15.28:0/3396342514 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7409d170 con 0x7f6d740b8eb0 2024-04-25T19:13:39.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.858+0000 7f6d7a35c640 1 --2- 172.21.15.28:0/3396342514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 0x7f6d7409bd30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6d740d2ac0 tx=0x7f6d7002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4473d67b103e3d5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.858+0000 7f6d7935a640 1 -- 172.21.15.28:0/3396342514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70037070 con 0x7f6d740b8eb0 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.858+0000 7f6d7935a640 1 -- 172.21.15.28:0/3396342514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d70032440 con 0x7f6d740b8eb0 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.858+0000 7f6d7935a640 1 -- 172.21.15.28:0/3396342514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d70032810 con 0x7f6d740b8eb0 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.859+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/3396342514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 msgr2=0x7f6d7409bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.859+0000 7f6d7c5c0640 1 --2- 172.21.15.28:0/3396342514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 0x7f6d7409bd30 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6d740d2ac0 tx=0x7f6d7002a1a0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.859+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/3396342514 shutdown_connections 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.859+0000 7f6d7c5c0640 1 --2- 172.21.15.28:0/3396342514 >> v2:172.21.15.53:3300/0 conn(0x7f6d7408cbb0 0x7f6d740d1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.859+0000 7f6d7c5c0640 1 --2- 172.21.15.28:0/3396342514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 0x7f6d7409bd30 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.859+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/3396342514 >> 172.21.15.28:0/3396342514 conn(0x7f6d740b5470 msgr2=0x7f6d740b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.859+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/3396342514 shutdown_connections 2024-04-25T19:13:39.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.859+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/3396342514 wait complete. 2024-04-25T19:13:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.861+0000 7f6d7c5c0640 1 Processor -- start 2024-04-25T19:13:39.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.861+0000 7f6d7c5c0640 1 -- start start 2024-04-25T19:13:39.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.862+0000 7f6d7c5c0640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7408cbb0 0x7f6d741e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.862+0000 7f6d7c5c0640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7409c270 0x7f6d740a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.862+0000 7f6d7c5c0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 0x7f6d740c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.862+0000 7f6d7c5c0640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6d740cf2e0 con 0x7f6d740b8eb0 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.862+0000 7f6d7c5c0640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f6d740cf000 con 0x7f6d7409c270 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.862+0000 7f6d7c5c0640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f6d740b50e0 con 0x7f6d7408cbb0 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 0x7f6d740c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 0x7f6d740c9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55302/0 (socket says 172.21.15.28:55302) 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 -- 172.21.15.28:0/2666271288 learned_addr learned my addr 172.21.15.28:0/2666271288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d79b5b640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7409c270 0x7f6d740a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7a35c640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7408cbb0 0x7f6d741e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 -- 172.21.15.28:0/2666271288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7408cbb0 msgr2=0x7f6d741e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f6d7408cbb0 0x7f6d741e6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 -- 172.21.15.28:0/2666271288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7409c270 msgr2=0x7f6d740a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7409c270 0x7f6d740a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 -- 172.21.15.28:0/2666271288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d741f7890 con 0x7f6d740b8eb0 2024-04-25T19:13:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7ab5d640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 0x7f6d740c9680 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c007e90 tx=0x7f6d6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c014420 con 0x7f6d740b8eb0 2024-04-25T19:13:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d6c019070 con 0x7f6d740b8eb0 2024-04-25T19:13:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d741f7d90 con 0x7f6d740b8eb0 2024-04-25T19:13:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c014800 con 0x7f6d740b8eb0 2024-04-25T19:13:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.863+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d741f8350 con 0x7f6d740b8eb0 2024-04-25T19:13:39.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.864+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6d741e6d90 con 0x7f6d740b8eb0 2024-04-25T19:13:39.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.865+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6d6c0149a0 con 0x7f6d740b8eb0 2024-04-25T19:13:39.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.865+0000 7f6d6affd640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d580744b0 0x7f6d580769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.866+0000 7f6d7a35c640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d580744b0 0x7f6d580769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.866+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f6d6c096720 con 0x7f6d740b8eb0 2024-04-25T19:13:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.866+0000 7f6d6affd640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f6d5807aa40 0x7f6d5807ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.866+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6d5807d5a0 con 0x7f6d5807aa40 2024-04-25T19:13:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.866+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f6d6c096ad0 con 0x7f6d740b8eb0 2024-04-25T19:13:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.866+0000 7f6d79b5b640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f6d5807aa40 0x7f6d5807ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.866+0000 7f6d7a35c640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d580744b0 0x7f6d580769c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6d70034000 tx=0x7f6d7002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.866+0000 7f6d79b5b640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f6d5807aa40 0x7f6d5807ced0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.867+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 <== osd.1 v2:172.21.15.28:6808/1898594658 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6d5807d5a0 con 0x7f6d5807aa40 2024-04-25T19:13:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.880+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 --> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6d7406a100 con 0x7f6d5807aa40 2024-04-25T19:13:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.880+0000 7f6d6affd640 1 -- 172.21.15.28:0/2666271288 <== osd.1 v2:172.21.15.28:6808/1898594658 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6d7406a100 con 0x7f6d5807aa40 2024-04-25T19:13:39.882 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:39.883 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T19:13:39.883 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:39.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f6d5807aa40 msgr2=0x7f6d5807ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6808/1898594658,v1:172.21.15.28:6809/1898594658] conn(0x7f6d5807aa40 0x7f6d5807ced0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d580744b0 msgr2=0x7f6d580769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f6d580744b0 0x7f6d580769c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6d70034000 tx=0x7f6d7002e000 comp rx=0 tx=0).stop 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 msgr2=0x7f6d740c9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 --2- 172.21.15.28:0/2666271288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d740b8eb0 0x7f6d740c9680 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c007e90 tx=0x7f6d6c00e4b0 comp rx=0 tx=0).stop 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7ab5d640 1 -- 172.21.15.28:0/2666271288 reap_dead start 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 shutdown_connections 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 >> 172.21.15.28:0/2666271288 conn(0x7f6d740b5470 msgr2=0x7f6d740b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 shutdown_connections 2024-04-25T19:13:39.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:39.881+0000 7f6d7c5c0640 1 -- 172.21.15.28:0/2666271288 wait complete. 2024-04-25T19:13:39.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T19:13:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.003+0000 7fb41bfba640 1 Processor -- start 2024-04-25T19:13:40.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.003+0000 7fb41bfba640 1 -- start start 2024-04-25T19:13:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb41bfba640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4140b3d00 0x7fb4140b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb41bfba640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4140b92a0 con 0x7fb4140b4650 2024-04-25T19:13:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb41bfba640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb41409cea0 con 0x7fb4140d2d10 2024-04-25T19:13:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb41bfba640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb4140d28d0 con 0x7fb4140b3d00 2024-04-25T19:13:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb419555640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4140b3d00 0x7fb4140b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb419555640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4140b3d00 0x7fb4140b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55306/0 (socket says 172.21.15.28:55306) 2024-04-25T19:13:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb419555640 1 -- 172.21.15.28:0/2622544219 learned_addr learned my addr 172.21.15.28:0/2622544219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb41a557640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb4140b4650 0x7fb41416a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50914/0 (socket says 172.21.15.28:50914) 2024-04-25T19:13:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb419555640 1 -- 172.21.15.28:0/2622544219 >> v1:172.21.15.53:6789/0 conn(0x7fb4140d2d10 legacy=0x7fb4140d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb419555640 1 -- 172.21.15.28:0/2622544219 >> v1:172.21.15.28:6789/0 conn(0x7fb4140b4650 legacy=0x7fb41416a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb418d54640 1 -- 172.21.15.28:0/2622544219 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3278796044 0 0) 0x7fb4140b92a0 con 0x7fb4140b4650 2024-04-25T19:13:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.005+0000 7fb419555640 1 -- 172.21.15.28:0/2622544219 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb41406e8e0 con 0x7fb4140b3d00 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.006+0000 7fb419555640 1 --2- 172.21.15.28:0/2622544219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb4140b4110 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb404009870 tx=0x7fb40402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b867a4a05736823d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.006+0000 7fb418d54640 1 -- 172.21.15.28:0/2622544219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb404037070 con 0x7fb4140b3d00 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.006+0000 7fb418d54640 1 -- 172.21.15.28:0/2622544219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb404032440 con 0x7fb4140b3d00 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.006+0000 7fb418d54640 1 -- 172.21.15.28:0/2622544219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb404032810 con 0x7fb4140b3d00 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.007+0000 7fb41bfba640 1 -- 172.21.15.28:0/2622544219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 msgr2=0x7fb4140b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.007+0000 7fb41bfba640 1 --2- 172.21.15.28:0/2622544219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb4140b4110 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fb404009870 tx=0x7fb40402a270 comp rx=0 tx=0).stop 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.007+0000 7fb41bfba640 1 -- 172.21.15.28:0/2622544219 shutdown_connections 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.007+0000 7fb41bfba640 1 --2- 172.21.15.28:0/2622544219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb4140b4110 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.007+0000 7fb41bfba640 1 -- 172.21.15.28:0/2622544219 >> 172.21.15.28:0/2622544219 conn(0x7fb4140d0450 msgr2=0x7fb4140d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.007+0000 7fb41bfba640 1 -- 172.21.15.28:0/2622544219 shutdown_connections 2024-04-25T19:13:40.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.007+0000 7fb41bfba640 1 -- 172.21.15.28:0/2622544219 wait complete. 2024-04-25T19:13:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.009+0000 7fb41bfba640 1 Processor -- start 2024-04-25T19:13:40.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.009+0000 7fb41bfba640 1 -- start start 2024-04-25T19:13:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb41bfba640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb41409d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb41bfba640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4140b4650 0x7fb41409ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb41bfba640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4140d2d10 0x7fb4141d6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb41bfba640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb41406ebc0 con 0x7fb4140b3d00 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb41bfba640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb41406e690 con 0x7fb4140d2d10 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb41bfba640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb4140d3120 con 0x7fb4140b4650 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb419d56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb41409d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb419d56640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb41409d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55310/0 (socket says 172.21.15.28:55310) 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.010+0000 7fb419d56640 1 -- 172.21.15.28:0/502805272 learned_addr learned my addr 172.21.15.28:0/502805272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb41a557640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4140d2d10 0x7fb4141d6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb419555640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4140b4650 0x7fb41409ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb419d56640 1 -- 172.21.15.28:0/502805272 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4140b4650 msgr2=0x7fb41409ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb419d56640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4140b4650 0x7fb41409ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb419d56640 1 -- 172.21.15.28:0/502805272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4140d2d10 msgr2=0x7fb4141d6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb419d56640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4140d2d10 0x7fb4141d6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb419d56640 1 -- 172.21.15.28:0/502805272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4141d8c10 con 0x7fb4140b3d00 2024-04-25T19:13:40.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb419d56640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb41409d740 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c3f0 tx=0x7fb40c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb40a7fc640 1 -- 172.21.15.28:0/502805272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c017070 con 0x7fb4140b3d00 2024-04-25T19:13:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb40a7fc640 1 -- 172.21.15.28:0/502805272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb40c0052f0 con 0x7fb4140b3d00 2024-04-25T19:13:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4141d9080 con 0x7fb4140b3d00 2024-04-25T19:13:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb40a7fc640 1 -- 172.21.15.28:0/502805272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40c003830 con 0x7fb4140b3d00 2024-04-25T19:13:40.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.011+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4141d9640 con 0x7fb4140b3d00 2024-04-25T19:13:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.012+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb414050740 con 0x7fb4140b3d00 2024-04-25T19:13:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.013+0000 7fb40a7fc640 1 -- 172.21.15.28:0/502805272 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb40c01a070 con 0x7fb4140b3d00 2024-04-25T19:13:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.013+0000 7fb40a7fc640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb3f80744b0 0x7fb3f80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.013+0000 7fb40a7fc640 1 -- 172.21.15.28:0/502805272 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7fb40c0943f0 con 0x7fb4140b3d00 2024-04-25T19:13:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.013+0000 7fb419555640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb3f80744b0 0x7fb3f80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.014+0000 7fb419555640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb3f80744b0 0x7fb3f80769c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb404034000 tx=0x7fb40402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.018+0000 7fb40a7fc640 1 -- 172.21.15.28:0/502805272 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb40c05edc0 con 0x7fb4140b3d00 2024-04-25T19:13:40.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.204+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fb41409dc80 con 0x7fb3f80744b0 2024-04-25T19:13:40.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.204+0000 7fb40a7fc640 1 -- 172.21.15.28:0/502805272 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb41409dc80 con 0x7fb3f80744b0 2024-04-25T19:13:40.207 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T19:13:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb3f80744b0 msgr2=0x7fb3f80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb3f80744b0 0x7fb3f80769c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb404034000 tx=0x7fb40402e000 comp rx=0 tx=0).stop 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 msgr2=0x7fb41409d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb41409d740 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c3f0 tx=0x7fb40c00c8b0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 shutdown_connections 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb3f80744b0 0x7fb3f80769c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4140d2d10 0x7fb4141d6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb4140b4650 0x7fb41409ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 --2- 172.21.15.28:0/502805272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4140b3d00 0x7fb41409d740 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 >> 172.21.15.28:0/502805272 conn(0x7fb4140d0450 msgr2=0x7fb4140d21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 shutdown_connections 2024-04-25T19:13:40.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.206+0000 7fb41bfba640 1 -- 172.21.15.28:0/502805272 wait complete. 2024-04-25T19:13:40.219 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T19:13:40.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T19:13:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.333+0000 7fa93a7ee640 1 Processor -- start 2024-04-25T19:13:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.333+0000 7fa93a7ee640 1 -- start start 2024-04-25T19:13:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa93a7ee640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa93406e5e0 0x7fa93416aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa93a7ee640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9340d12e0 con 0x7fa9340d5690 2024-04-25T19:13:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa93a7ee640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9340d15c0 con 0x7fa9340b47d0 2024-04-25T19:13:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa93a7ee640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa9340d4bb0 con 0x7fa93406e5e0 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa9337fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa9340d5690 0x7fa93406ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50928/0 (socket says 172.21.15.28:50928) 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa9337fe640 1 -- 172.21.15.28:0/2452512219 learned_addr learned my addr 172.21.15.28:0/2452512219 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa938d8b640 1 --2- 172.21.15.28:0/2452512219 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa93406e5e0 0x7fa93416aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa932ffd640 1 -- 172.21.15.28:0/2452512219 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2337242408 0 0) 0x7fa9340d12e0 con 0x7fa9340d5690 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa932ffd640 1 -- 172.21.15.28:0/2452512219 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa920003520 con 0x7fa9340d5690 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa932ffd640 1 -- 172.21.15.28:0/2452512219 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1221006287 0 0) 0x7fa9340d15c0 con 0x7fa9340b47d0 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa938d8b640 1 -- 172.21.15.28:0/2452512219 >> v1:172.21.15.53:6789/0 conn(0x7fa9340b47d0 legacy=0x7fa9341601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa938d8b640 1 -- 172.21.15.28:0/2452512219 >> v1:172.21.15.28:6789/0 conn(0x7fa9340d5690 legacy=0x7fa93406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.335+0000 7fa938d8b640 1 -- 172.21.15.28:0/2452512219 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9340d4e90 con 0x7fa93406e5e0 2024-04-25T19:13:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.336+0000 7fa938d8b640 1 --2- 172.21.15.28:0/2452512219 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa93406e5e0 0x7fa93416aec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009dc0 tx=0x7fa92c02a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e696cda5d2df991 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.336+0000 7fa932ffd640 1 -- 172.21.15.28:0/2452512219 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c035020 con 0x7fa93406e5e0 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa932ffd640 1 -- 172.21.15.28:0/2452512219 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa92c007540 con 0x7fa93406e5e0 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa932ffd640 1 -- 172.21.15.28:0/2452512219 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa924001690 con 0x7fa93406e5e0 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa93a7ee640 1 -- 172.21.15.28:0/2452512219 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa93406e5e0 msgr2=0x7fa93416aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa93a7ee640 1 --2- 172.21.15.28:0/2452512219 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa93406e5e0 0x7fa93416aec0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009dc0 tx=0x7fa92c02a640 comp rx=0 tx=0).stop 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa93a7ee640 1 -- 172.21.15.28:0/2452512219 shutdown_connections 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa93a7ee640 1 --2- 172.21.15.28:0/2452512219 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa93406e5e0 0x7fa93416aec0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa93a7ee640 1 -- 172.21.15.28:0/2452512219 >> 172.21.15.28:0/2452512219 conn(0x7fa9340d2c60 msgr2=0x7fa9340d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa93a7ee640 1 -- 172.21.15.28:0/2452512219 shutdown_connections 2024-04-25T19:13:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.337+0000 7fa93a7ee640 1 -- 172.21.15.28:0/2452512219 wait complete. 2024-04-25T19:13:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.339+0000 7fa93a7ee640 1 Processor -- start 2024-04-25T19:13:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.339+0000 7fa93a7ee640 1 -- start start 2024-04-25T19:13:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.340+0000 7fa93a7ee640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9340b47d0 0x7fa9340ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.340+0000 7fa93a7ee640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9340d5690 0x7fa9341f9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.340+0000 7fa93a7ee640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9341f9660 0x7fa93406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.340+0000 7fa93a7ee640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9340b4e80 con 0x7fa9341f9660 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.340+0000 7fa93a7ee640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fa9340b4d00 con 0x7fa9340d5690 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.340+0000 7fa93a7ee640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fa9340b5000 con 0x7fa9340b47d0 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9341f9660 0x7fa93406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9341f9660 0x7fa93406e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55322/0 (socket says 172.21.15.28:55322) 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 -- 172.21.15.28:0/81426655 learned_addr learned my addr 172.21.15.28:0/81426655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa933fff640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9340b47d0 0x7fa9340ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa9337fe640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9340d5690 0x7fa9341f9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 -- 172.21.15.28:0/81426655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9340b47d0 msgr2=0x7fa9340ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fa9340b47d0 0x7fa9340ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 -- 172.21.15.28:0/81426655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9340d5690 msgr2=0x7fa9341f9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9340d5690 0x7fa9341f9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 -- 172.21.15.28:0/81426655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9341f9fc0 con 0x7fa9341f9660 2024-04-25T19:13:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa938d8b640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9341f9660 0x7fa93406e0b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009ee0 tx=0x7fa92c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c03e030 con 0x7fa9341f9660 2024-04-25T19:13:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9341f9ce0 con 0x7fa9341f9660 2024-04-25T19:13:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa92c004040 con 0x7fa9341f9660 2024-04-25T19:13:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92c035040 con 0x7fa9341f9660 2024-04-25T19:13:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.341+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9340c9a60 con 0x7fa9341f9660 2024-04-25T19:13:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.342+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa93406a100 con 0x7fa9341f9660 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.343+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa92c041030 con 0x7fa9341f9660 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.343+0000 7fa930ff9640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9140744e0 0x7fa9140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.344+0000 7fa933fff640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9140744e0 0x7fa9140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.344+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7fa92c0b59e0 con 0x7fa9341f9660 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.344+0000 7fa930ff9640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7fa91407aa70 0x7fa91407cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.344+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa92c037e50 con 0x7fa91407aa70 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.344+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fa92c0ba2c0 con 0x7fa9341f9660 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.344+0000 7fa9337fe640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7fa91407aa70 0x7fa91407cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.344+0000 7fa933fff640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9140744e0 0x7fa9140769f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa91c000f60 tx=0x7fa91c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.344+0000 7fa9337fe640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7fa91407aa70 0x7fa91407cf00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.345+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 <== osd.2 v2:172.21.15.28:6816/3410596961 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa92c037e50 con 0x7fa91407aa70 2024-04-25T19:13:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.358+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 --> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa9341f9a70 con 0x7fa91407aa70 2024-04-25T19:13:40.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.358+0000 7fa930ff9640 1 -- 172.21.15.28:0/81426655 <== osd.2 v2:172.21.15.28:6816/3410596961 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa9341f9a70 con 0x7fa91407aa70 2024-04-25T19:13:40.360 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:40.360 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T19:13:40.360 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7fa91407aa70 msgr2=0x7fa91407cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6816/3410596961,v1:172.21.15.28:6817/3410596961] conn(0x7fa91407aa70 0x7fa91407cf00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9140744e0 msgr2=0x7fa9140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fa9140744e0 0x7fa9140769f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa91c000f60 tx=0x7fa91c009040 comp rx=0 tx=0).stop 2024-04-25T19:13:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9341f9660 msgr2=0x7fa93406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 --2- 172.21.15.28:0/81426655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9341f9660 0x7fa93406e0b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009ee0 tx=0x7fa92c004a40 comp rx=0 tx=0).stop 2024-04-25T19:13:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa938d8b640 1 -- 172.21.15.28:0/81426655 reap_dead start 2024-04-25T19:13:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 shutdown_connections 2024-04-25T19:13:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 >> 172.21.15.28:0/81426655 conn(0x7fa9340d2c60 msgr2=0x7fa93415dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 shutdown_connections 2024-04-25T19:13:40.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.359+0000 7fa93a7ee640 1 -- 172.21.15.28:0/81426655 wait complete. 2024-04-25T19:13:40.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T19:13:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.481+0000 7f27ed726640 1 Processor -- start 2024-04-25T19:13:40.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.482+0000 7f27ed726640 1 -- start start 2024-04-25T19:13:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27ed726640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b79f0 0x7f27e80b7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27ed726640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27e80b9920 0x7f27e80b9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27ed726640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e8160720 con 0x7f27e8160310 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27ed726640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f27e80b8340 con 0x7f27e80b9920 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27ed726640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f27e80d0f40 con 0x7f27e80b79f0 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27e67fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b79f0 0x7f27e80b7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27e77fe640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27e80b9920 0x7f27e80b9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27e67fc640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b79f0 0x7f27e80b7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:58720/0 (socket says 172.21.15.28:58720) 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.483+0000 7f27e67fc640 1 -- 172.21.15.28:0/3419145535 learned_addr learned my addr 172.21.15.28:0/3419145535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.484+0000 7f27e67fc640 1 -- 172.21.15.28:0/3419145535 >> v2:172.21.15.53:3300/0 conn(0x7f27e80b9920 msgr2=0x7f27e80b9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.484+0000 7f27e67fc640 1 --2- 172.21.15.28:0/3419145535 >> v2:172.21.15.53:3300/0 conn(0x7f27e80b9920 0x7f27e80b9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.484+0000 7f27e67fc640 1 -- 172.21.15.28:0/3419145535 >> v1:172.21.15.53:6789/0 conn(0x7f27e8160310 legacy=0x7f27e80d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.484+0000 7f27e67fc640 1 -- 172.21.15.28:0/3419145535 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e80d1220 con 0x7f27e80b79f0 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.484+0000 7f27e67fc640 1 --2- 172.21.15.28:0/3419145535 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b79f0 0x7f27e80b7e00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009870 tx=0x7f27dc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba7168a66d584010 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27e5ffb640 1 -- 172.21.15.28:0/3419145535 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc037070 con 0x7f27e80b79f0 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27e5ffb640 1 -- 172.21.15.28:0/3419145535 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27dc032440 con 0x7f27e80b79f0 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27e5ffb640 1 -- 172.21.15.28:0/3419145535 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27dc032810 con 0x7f27e80b79f0 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27ed726640 1 -- 172.21.15.28:0/3419145535 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b79f0 msgr2=0x7f27e80b7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27ed726640 1 --2- 172.21.15.28:0/3419145535 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b79f0 0x7f27e80b7e00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009870 tx=0x7f27dc02a270 comp rx=0 tx=0).stop 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27ed726640 1 -- 172.21.15.28:0/3419145535 shutdown_connections 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27ed726640 1 --2- 172.21.15.28:0/3419145535 >> v2:172.21.15.53:3300/0 conn(0x7f27e80b9920 0x7f27e80b9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27ed726640 1 --2- 172.21.15.28:0/3419145535 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b79f0 0x7f27e80b7e00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27ed726640 1 -- 172.21.15.28:0/3419145535 >> 172.21.15.28:0/3419145535 conn(0x7f27e80d2a50 msgr2=0x7f27e80d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27ed726640 1 -- 172.21.15.28:0/3419145535 shutdown_connections 2024-04-25T19:13:40.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.485+0000 7f27ed726640 1 -- 172.21.15.28:0/3419145535 wait complete. 2024-04-25T19:13:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.487+0000 7f27ed726640 1 Processor -- start 2024-04-25T19:13:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.487+0000 7f27ed726640 1 -- start start 2024-04-25T19:13:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27ed726640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e80b79f0 0x7f27e81f5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27ed726640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b9920 0x7f27e814af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27ed726640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e8160310 0x7f27e80ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27ed726640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27e80d0bf0 con 0x7f27e8160310 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27ed726640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f27e80b9400 con 0x7f27e80b79f0 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27ed726640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f27e80cc950 con 0x7f27e80b9920 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e8160310 0x7f27e80ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e8160310 0x7f27e80ccc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55326/0 (socket says 172.21.15.28:55326) 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 -- 172.21.15.28:0/2730105470 learned_addr learned my addr 172.21.15.28:0/2730105470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e67fc640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b9920 0x7f27e814af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e6ffd640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e80b79f0 0x7f27e81f5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 -- 172.21.15.28:0/2730105470 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b9920 msgr2=0x7f27e814af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b9920 0x7f27e814af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 -- 172.21.15.28:0/2730105470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e80b79f0 msgr2=0x7f27e81f5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e80b79f0 0x7f27e81f5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 -- 172.21.15.28:0/2730105470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e8149ed0 con 0x7f27e8160310 2024-04-25T19:13:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.489+0000 7f27e77fe640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e8160310 0x7f27e80ccc10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f27e000e820 tx=0x7f27e000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.490+0000 7f27cf7fe640 1 -- 172.21.15.28:0/2730105470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0017070 con 0x7f27e8160310 2024-04-25T19:13:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.490+0000 7f27cf7fe640 1 -- 172.21.15.28:0/2730105470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e000ee70 con 0x7f27e8160310 2024-04-25T19:13:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.490+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e814a1b0 con 0x7f27e8160310 2024-04-25T19:13:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.490+0000 7f27cf7fe640 1 -- 172.21.15.28:0/2730105470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e000cb60 con 0x7f27e8160310 2024-04-25T19:13:40.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.490+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e814a770 con 0x7f27e8160310 2024-04-25T19:13:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.491+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e8050740 con 0x7f27e8160310 2024-04-25T19:13:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.491+0000 7f27cf7fe640 1 -- 172.21.15.28:0/2730105470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f27e001a070 con 0x7f27e8160310 2024-04-25T19:13:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.492+0000 7f27cf7fe640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27d40744e0 0x7f27d40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.492+0000 7f27cf7fe640 1 -- 172.21.15.28:0/2730105470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f27e0095300 con 0x7f27e8160310 2024-04-25T19:13:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.492+0000 7f27e6ffd640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27d40744e0 0x7f27d40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.492+0000 7f27e6ffd640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27d40744e0 0x7f27d40769f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f27d8001ca0 tx=0x7f27d8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.496+0000 7f27cf7fe640 1 -- 172.21.15.28:0/2730105470 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f27e005fbb0 con 0x7f27e8160310 2024-04-25T19:13:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.681+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f27e815e1b0 con 0x7f27d40744e0 2024-04-25T19:13:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.681+0000 7f27cf7fe640 1 -- 172.21.15.28:0/2730105470 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f27e815e1b0 con 0x7f27d40744e0 2024-04-25T19:13:40.683 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T19:13:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27d40744e0 msgr2=0x7f27d40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27d40744e0 0x7f27d40769f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f27d8001ca0 tx=0x7f27d8009040 comp rx=0 tx=0).stop 2024-04-25T19:13:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e8160310 msgr2=0x7f27e80ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e8160310 0x7f27e80ccc10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f27e000e820 tx=0x7f27e000ec20 comp rx=0 tx=0).stop 2024-04-25T19:13:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 shutdown_connections 2024-04-25T19:13:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f27d40744e0 0x7f27d40769f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27e8160310 0x7f27e80ccc10 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f27e80b9920 0x7f27e814af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 --2- 172.21.15.28:0/2730105470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e80b79f0 0x7f27e81f5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 >> 172.21.15.28:0/2730105470 conn(0x7f27e80d2a50 msgr2=0x7f27e815d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 shutdown_connections 2024-04-25T19:13:40.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.683+0000 7f27ed726640 1 -- 172.21.15.28:0/2730105470 wait complete. 2024-04-25T19:13:40.696 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T19:13:40.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T19:13:40.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.810+0000 7fb61c461640 1 Processor -- start 2024-04-25T19:13:40.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.810+0000 7fb61c461640 1 -- start start 2024-04-25T19:13:40.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb61c461640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6140b41f0 0x7fb6140b4600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb61c461640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 0x7fb6140d2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb61c461640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6140d35b0 con 0x7fb6141576f0 2024-04-25T19:13:40.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb61c461640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fb6140b2740 con 0x7fb6140b41f0 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb61c461640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb6140b2a20 con 0x7fb6140d2800 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb6199fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6140b41f0 0x7fb6140b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb6199fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6140b41f0 0x7fb6140b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58506/0 (socket says 172.21.15.28:58506) 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb61a1fd640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb6141576f0 0x7fb6140b3500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38892/0 (socket says 172.21.15.28:38892) 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb6199fc640 1 -- 172.21.15.28:0/1905234960 learned_addr learned my addr 172.21.15.28:0/1905234960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb61a9fe640 1 --2- 172.21.15.28:0/1905234960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 0x7fb6140d2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb6199fc640 1 -- 172.21.15.28:0/1905234960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 msgr2=0x7fb6140d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb6199fc640 1 --2- 172.21.15.28:0/1905234960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 0x7fb6140d2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb6199fc640 1 -- 172.21.15.28:0/1905234960 >> v1:172.21.15.28:6789/0 conn(0x7fb6141576f0 legacy=0x7fb6140b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb6191fb640 1 -- 172.21.15.28:0/1905234960 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395098487 0 0) 0x7fb6140d35b0 con 0x7fb6141576f0 2024-04-25T19:13:40.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.812+0000 7fb6199fc640 1 -- 172.21.15.28:0/1905234960 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6140b2d00 con 0x7fb6140b41f0 2024-04-25T19:13:40.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.813+0000 7fb6199fc640 1 --2- 172.21.15.28:0/1905234960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 0x7fb6140b4600 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fb608009870 tx=0x7fb60802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27e44917599aef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.813+0000 7fb6191fb640 1 -- 172.21.15.28:0/1905234960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608037070 con 0x7fb6140b41f0 2024-04-25T19:13:40.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.813+0000 7fb6191fb640 1 -- 172.21.15.28:0/1905234960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb608032440 con 0x7fb6140b41f0 2024-04-25T19:13:40.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.813+0000 7fb6191fb640 1 -- 172.21.15.28:0/1905234960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb608032810 con 0x7fb6140b41f0 2024-04-25T19:13:40.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.814+0000 7fb61c461640 1 -- 172.21.15.28:0/1905234960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 msgr2=0x7fb6140b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.814+0000 7fb61c461640 1 --2- 172.21.15.28:0/1905234960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 0x7fb6140b4600 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fb608009870 tx=0x7fb60802a1c0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.814+0000 7fb61c461640 1 -- 172.21.15.28:0/1905234960 shutdown_connections 2024-04-25T19:13:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.814+0000 7fb61c461640 1 --2- 172.21.15.28:0/1905234960 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 0x7fb6140d2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.814+0000 7fb61c461640 1 --2- 172.21.15.28:0/1905234960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 0x7fb6140b4600 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.814+0000 7fb61c461640 1 -- 172.21.15.28:0/1905234960 >> 172.21.15.28:0/1905234960 conn(0x7fb6140b7ee0 msgr2=0x7fb6140b7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.814+0000 7fb61c461640 1 -- 172.21.15.28:0/1905234960 shutdown_connections 2024-04-25T19:13:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.814+0000 7fb61c461640 1 -- 172.21.15.28:0/1905234960 wait complete. 2024-04-25T19:13:40.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.816+0000 7fb61c461640 1 Processor -- start 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.816+0000 7fb61c461640 1 -- start start 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61c461640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 0x7fb6140d0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61c461640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 0x7fb614146230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61c461640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6141576f0 0x7fb6140b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61c461640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb6140d3150 con 0x7fb6140b41f0 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61c461640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fb6140b4b40 con 0x7fb6141576f0 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61c461640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fb6140d32d0 con 0x7fb6140d2800 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61a1fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 0x7fb6140d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61a1fd640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 0x7fb6140d0830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58516/0 (socket says 172.21.15.28:58516) 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61a1fd640 1 -- 172.21.15.28:0/541140168 learned_addr learned my addr 172.21.15.28:0/541140168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb6199fc640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 0x7fb614146230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.817+0000 7fb61a9fe640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6141576f0 0x7fb6140b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61a1fd640 1 -- 172.21.15.28:0/541140168 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 msgr2=0x7fb614146230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61a1fd640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 0x7fb614146230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61a1fd640 1 -- 172.21.15.28:0/541140168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6141576f0 msgr2=0x7fb6140b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61a1fd640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6141576f0 0x7fb6140b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61a1fd640 1 -- 172.21.15.28:0/541140168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6140cef00 con 0x7fb6140b41f0 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb6199fc640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fb6140d2800 0x7fb614146230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61a9fe640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6141576f0 0x7fb6140b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61a1fd640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 0x7fb6140d0830 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb60400c7b0 tx=0x7fb60400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb604018070 con 0x7fb6140b41f0 2024-04-25T19:13:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb60400a640 con 0x7fb6140b41f0 2024-04-25T19:13:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6140cf1e0 con 0x7fb6140b41f0 2024-04-25T19:13:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60400aa30 con 0x7fb6140b41f0 2024-04-25T19:13:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.818+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6140cf7a0 con 0x7fb6140b41f0 2024-04-25T19:13:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.819+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb6140cffa0 con 0x7fb6140b41f0 2024-04-25T19:13:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.820+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb60401b070 con 0x7fb6140b41f0 2024-04-25T19:13:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.820+0000 7fb612ffd640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6000744b0 0x7fb6000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.820+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7fb604094d90 con 0x7fb6140b41f0 2024-04-25T19:13:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.820+0000 7fb6199fc640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6000744b0 0x7fb6000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.820+0000 7fb612ffd640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7fb60007aa40 0x7fb60007ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.820+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb60401fe50 con 0x7fb60007aa40 2024-04-25T19:13:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.820+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fb604095140 con 0x7fb6140b41f0 2024-04-25T19:13:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.821+0000 7fb61a9fe640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7fb60007aa40 0x7fb60007ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.821+0000 7fb6199fc640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6000744b0 0x7fb6000769c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb608034000 tx=0x7fb60802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.821+0000 7fb61a9fe640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7fb60007aa40 0x7fb60007ced0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.822+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 <== osd.3 v2:172.21.15.28:6818/1928233777 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb60401fe50 con 0x7fb60007aa40 2024-04-25T19:13:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 --> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb614074900 con 0x7fb60007aa40 2024-04-25T19:13:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb612ffd640 1 -- 172.21.15.28:0/541140168 <== osd.3 v2:172.21.15.28:6818/1928233777 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb614074900 con 0x7fb60007aa40 2024-04-25T19:13:40.837 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:40.837 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T19:13:40.837 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7fb60007aa40 msgr2=0x7fb60007ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb61c461640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6818/1928233777,v1:172.21.15.28:6820/1928233777] conn(0x7fb60007aa40 0x7fb60007ced0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6000744b0 msgr2=0x7fb6000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb61c461640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fb6000744b0 0x7fb6000769c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fb608034000 tx=0x7fb60802e000 comp rx=0 tx=0).stop 2024-04-25T19:13:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 msgr2=0x7fb6140d0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb61c461640 1 --2- 172.21.15.28:0/541140168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6140b41f0 0x7fb6140d0830 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb60400c7b0 tx=0x7fb60400cc70 comp rx=0 tx=0).stop 2024-04-25T19:13:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.835+0000 7fb61a9fe640 1 -- 172.21.15.28:0/541140168 reap_dead start 2024-04-25T19:13:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.836+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 shutdown_connections 2024-04-25T19:13:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.836+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 >> 172.21.15.28:0/541140168 conn(0x7fb6140b7ee0 msgr2=0x7fb61415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.836+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 shutdown_connections 2024-04-25T19:13:40.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.836+0000 7fb61c461640 1 -- 172.21.15.28:0/541140168 wait complete. 2024-04-25T19:13:40.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T19:13:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.959+0000 7f26bc534640 1 Processor -- start 2024-04-25T19:13:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.959+0000 7f26bc534640 1 -- start start 2024-04-25T19:13:40.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26bc534640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26b4164990 0x7f26b415dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26bc534640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b40b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26bc534640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26b4161210 con 0x7f26b41606f0 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26bc534640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f26b40b7fb0 con 0x7f26b4164990 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26bc534640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f26b41614a0 con 0x7f26b40b7660 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26ba2d0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26b4164990 0x7f26b415dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26baad1640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f26b41606f0 0x7f26b4160b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38894/0 (socket says 172.21.15.28:38894) 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26ba2d0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26b4164990 0x7f26b415dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58522/0 (socket says 172.21.15.28:58522) 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.960+0000 7f26baad1640 1 -- 172.21.15.28:0/3188299853 learned_addr learned my addr 172.21.15.28:0/3188299853 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.961+0000 7f26b9acf640 1 --2- 172.21.15.28:0/3188299853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b40b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.961+0000 7f26ba2d0640 1 -- 172.21.15.28:0/3188299853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 msgr2=0x7f26b40b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.961+0000 7f26ba2d0640 1 --2- 172.21.15.28:0/3188299853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b40b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.961+0000 7f26ba2d0640 1 -- 172.21.15.28:0/3188299853 >> v1:172.21.15.28:6789/0 conn(0x7f26b41606f0 legacy=0x7f26b4160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.961+0000 7f26ba2d0640 1 -- 172.21.15.28:0/3188299853 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b4163eb0 con 0x7f26b4164990 2024-04-25T19:13:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.961+0000 7f26b9acf640 1 --2- 172.21.15.28:0/3188299853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b40b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.961+0000 7f26ba2d0640 1 --2- 172.21.15.28:0/3188299853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b4164990 0x7f26b415dcb0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f26b00097a0 tx=0x7f26b002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d81319a03ce330 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26b92ce640 1 -- 172.21.15.28:0/3188299853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b0037070 con 0x7f26b4164990 2024-04-25T19:13:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26b92ce640 1 -- 172.21.15.28:0/3188299853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26b0032440 con 0x7f26b4164990 2024-04-25T19:13:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26b92ce640 1 -- 172.21.15.28:0/3188299853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b0032810 con 0x7f26b4164990 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26bc534640 1 -- 172.21.15.28:0/3188299853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b4164990 msgr2=0x7f26b415dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26bc534640 1 --2- 172.21.15.28:0/3188299853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b4164990 0x7f26b415dcb0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f26b00097a0 tx=0x7f26b002a180 comp rx=0 tx=0).stop 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26bc534640 1 -- 172.21.15.28:0/3188299853 shutdown_connections 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26bc534640 1 --2- 172.21.15.28:0/3188299853 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b40b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26bc534640 1 --2- 172.21.15.28:0/3188299853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b4164990 0x7f26b415dcb0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26bc534640 1 -- 172.21.15.28:0/3188299853 >> 172.21.15.28:0/3188299853 conn(0x7f26b40d2840 msgr2=0x7f26b40d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26bc534640 1 -- 172.21.15.28:0/3188299853 shutdown_connections 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.962+0000 7f26bc534640 1 -- 172.21.15.28:0/3188299853 wait complete. 2024-04-25T19:13:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.964+0000 7f26bc534640 1 Processor -- start 2024-04-25T19:13:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.964+0000 7f26bc534640 1 -- start start 2024-04-25T19:13:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.965+0000 7f26bc534640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b415c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.965+0000 7f26bc534640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41606f0 0x7f26b40cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.965+0000 7f26bc534640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b4164990 0x7f26b40d6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.965+0000 7f26bc534640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f26b40b45d0 con 0x7f26b41606f0 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.965+0000 7f26bc534640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f26b4164470 con 0x7f26b4164990 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.965+0000 7f26bc534640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f26b40b4750 con 0x7f26b40b7660 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41606f0 0x7f26b40cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41606f0 0x7f26b40cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58524/0 (socket says 172.21.15.28:58524) 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 -- 172.21.15.28:0/270869660 learned_addr learned my addr 172.21.15.28:0/270869660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26baad1640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b4164990 0x7f26b40d6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26ba2d0640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b415c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 -- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 msgr2=0x7f26b415c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b415c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 -- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b4164990 msgr2=0x7f26b40d6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b4164990 0x7f26b40d6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 -- 172.21.15.28:0/270869660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b40cd120 con 0x7f26b41606f0 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26baad1640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b4164990 0x7f26b40d6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26ba2d0640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b415c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26b9acf640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41606f0 0x7f26b40cd640 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c7f0 tx=0x7f26a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26aaffd640 1 -- 172.21.15.28:0/270869660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a4015070 con 0x7f26b41606f0 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26aaffd640 1 -- 172.21.15.28:0/270869660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26a4007820 con 0x7f26b41606f0 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b40c9890 con 0x7f26b41606f0 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26aaffd640 1 -- 172.21.15.28:0/270869660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a4007bd0 con 0x7f26b41606f0 2024-04-25T19:13:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.966+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b40c9e50 con 0x7f26b41606f0 2024-04-25T19:13:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.967+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26b40cce90 con 0x7f26b41606f0 2024-04-25T19:13:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.968+0000 7f26aaffd640 1 -- 172.21.15.28:0/270869660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f26a4018070 con 0x7f26b41606f0 2024-04-25T19:13:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.968+0000 7f26aaffd640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26980744b0 0x7f26980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.969+0000 7f26aaffd640 1 -- 172.21.15.28:0/270869660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f26a40952f0 con 0x7f26b41606f0 2024-04-25T19:13:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.969+0000 7f26ba2d0640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26980744b0 0x7f26980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.969+0000 7f26ba2d0640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26980744b0 0x7f26980769c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f26b0034000 tx=0x7f26b002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:40.973+0000 7f26aaffd640 1 -- 172.21.15.28:0/270869660 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26a405fb90 con 0x7f26b41606f0 2024-04-25T19:13:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.162+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f26b4050740 con 0x7f26980744b0 2024-04-25T19:13:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.162+0000 7f26aaffd640 1 -- 172.21.15.28:0/270869660 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f26b4050740 con 0x7f26980744b0 2024-04-25T19:13:41.165 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T19:13:41.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26980744b0 msgr2=0x7f26980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26980744b0 0x7f26980769c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f26b0034000 tx=0x7f26b002e000 comp rx=0 tx=0).stop 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41606f0 msgr2=0x7f26b40cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41606f0 0x7f26b40cd640 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f26a400c7f0 tx=0x7f26a400ccb0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 shutdown_connections 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f26980744b0 0x7f26980769c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f26b4164990 0x7f26b40d6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26b41606f0 0x7f26b40cd640 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 --2- 172.21.15.28:0/270869660 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f26b40b7660 0x7f26b415c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.164+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 >> 172.21.15.28:0/270869660 conn(0x7f26b40d2840 msgr2=0x7f26b40bb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.165+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 shutdown_connections 2024-04-25T19:13:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.165+0000 7f26bc534640 1 -- 172.21.15.28:0/270869660 wait complete. 2024-04-25T19:13:41.177 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T19:13:41.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T19:13:41.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.291+0000 7fd69993f640 1 Processor -- start 2024-04-25T19:13:41.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.291+0000 7fd69993f640 1 -- start start 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd69993f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd6941601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd69993f640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6940d5690 0x7fd69406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd69993f640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6940b4d00 0x7fd69406e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd69993f640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fd6940d12e0 con 0x7fd6940d5690 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd69993f640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fd6940d15c0 con 0x7fd6940b4d00 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd69993f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd6940d4bb0 con 0x7fd6940b47d0 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6927fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6940d5690 0x7fd69406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6927fc640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6940d5690 0x7fd69406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58526/0 (socket says 172.21.15.28:58526) 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6927fc640 1 -- 172.21.15.28:0/2107319029 learned_addr learned my addr 172.21.15.28:0/2107319029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd692ffd640 1 --2- 172.21.15.28:0/2107319029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd6941601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6937fe640 1 --2- 172.21.15.28:0/2107319029 >> v2:172.21.15.53:3300/0 conn(0x7fd6940b4d00 0x7fd69406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6927fc640 1 -- 172.21.15.28:0/2107319029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 msgr2=0x7fd6941601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6927fc640 1 --2- 172.21.15.28:0/2107319029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd6941601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6927fc640 1 -- 172.21.15.28:0/2107319029 >> v2:172.21.15.53:3300/0 conn(0x7fd6940b4d00 msgr2=0x7fd69406e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6927fc640 1 --2- 172.21.15.28:0/2107319029 >> v2:172.21.15.53:3300/0 conn(0x7fd6940b4d00 0x7fd69406e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6927fc640 1 -- 172.21.15.28:0/2107319029 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6940d4e90 con 0x7fd6940d5690 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd6937fe640 1 --2- 172.21.15.28:0/2107319029 >> v2:172.21.15.53:3300/0 conn(0x7fd6940b4d00 0x7fd69406e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.293+0000 7fd692ffd640 1 --2- 172.21.15.28:0/2107319029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd6941601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.294+0000 7fd6927fc640 1 --2- 172.21.15.28:0/2107319029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 0x7fd69406ded0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd6940d17d0 tx=0x7fd67c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d374e489f0d1042 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.294+0000 7fd691ffb640 1 -- 172.21.15.28:0/2107319029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c037070 con 0x7fd6940d5690 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.294+0000 7fd691ffb640 1 -- 172.21.15.28:0/2107319029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67c032440 con 0x7fd6940d5690 2024-04-25T19:13:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.294+0000 7fd691ffb640 1 -- 172.21.15.28:0/2107319029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c032810 con 0x7fd6940d5690 2024-04-25T19:13:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 -- 172.21.15.28:0/2107319029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 msgr2=0x7fd69406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 --2- 172.21.15.28:0/2107319029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 0x7fd69406ded0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fd6940d17d0 tx=0x7fd67c02a1c0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 -- 172.21.15.28:0/2107319029 shutdown_connections 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 --2- 172.21.15.28:0/2107319029 >> v2:172.21.15.53:3300/0 conn(0x7fd6940b4d00 0x7fd69406e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 --2- 172.21.15.28:0/2107319029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 0x7fd69406ded0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 --2- 172.21.15.28:0/2107319029 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd6941601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 -- 172.21.15.28:0/2107319029 >> 172.21.15.28:0/2107319029 conn(0x7fd6940d2c60 msgr2=0x7fd6940d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 -- 172.21.15.28:0/2107319029 shutdown_connections 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.295+0000 7fd69993f640 1 -- 172.21.15.28:0/2107319029 wait complete. 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.297+0000 7fd69993f640 1 Processor -- start 2024-04-25T19:13:41.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.297+0000 7fd69993f640 1 -- start start 2024-04-25T19:13:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd69993f640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd694151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd69993f640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6940b4d00 0x7fd6941f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd69993f640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 0x7fd6940b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd69993f640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd6940d3780 con 0x7fd6940d5690 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd69993f640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd694160730 con 0x7fd6940b4d00 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd69993f640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd6940d3900 con 0x7fd6940b47d0 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd6937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 0x7fd6940b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd6937fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 0x7fd6940b6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58534/0 (socket says 172.21.15.28:58534) 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.298+0000 7fd6937fe640 1 -- 172.21.15.28:0/781810592 learned_addr learned my addr 172.21.15.28:0/781810592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd692ffd640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd694151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd6927fc640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6940b4d00 0x7fd6941f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd6937fe640 1 -- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 msgr2=0x7fd694151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd6937fe640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd694151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd6937fe640 1 -- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6940b4d00 msgr2=0x7fd6941f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd6937fe640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6940b4d00 0x7fd6941f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd6937fe640 1 -- 172.21.15.28:0/781810592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69415bf70 con 0x7fd6940d5690 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd692ffd640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd6940b47d0 0x7fd694151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd6927fc640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6940b4d00 0x7fd6941f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd6937fe640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 0x7fd6940b6790 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fd68400cd80 tx=0x7fd68400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68400f680 con 0x7fd6940d5690 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd68400f840 con 0x7fd6940d5690 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69415c250 con 0x7fd6940d5690 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68400fc30 con 0x7fd6940d5690 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.299+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd69414ed70 con 0x7fd6940d5690 2024-04-25T19:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.300+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd6940749b0 con 0x7fd6940d5690 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.301+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd6840213f0 con 0x7fd6940d5690 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.301+0000 7fd68bfff640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6740744b0 0x7fd6740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.301+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7fd684095d80 con 0x7fd6940d5690 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.301+0000 7fd692ffd640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6740744b0 0x7fd6740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.301+0000 7fd68bfff640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7fd67407aa40 0x7fd67407ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.301+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd67407d5a0 con 0x7fd67407aa40 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.301+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7fd684098480 con 0x7fd6940d5690 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.302+0000 7fd692ffd640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6740744b0 0x7fd6740769c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd68c006fd0 tx=0x7fd68c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.302+0000 7fd6927fc640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7fd67407aa40 0x7fd67407ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.302+0000 7fd6927fc640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7fd67407aa40 0x7fd67407ced0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.303+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 <== osd.4 v2:172.21.15.53:6824/2055146181 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd67407d5a0 con 0x7fd67407aa40 2024-04-25T19:13:41.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.316+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 --> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd69415bd20 con 0x7fd67407aa40 2024-04-25T19:13:41.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd68bfff640 1 -- 172.21.15.28:0/781810592 <== osd.4 v2:172.21.15.53:6824/2055146181 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd69415bd20 con 0x7fd67407aa40 2024-04-25T19:13:41.319 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:41.319 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T19:13:41.319 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7fd67407aa40 msgr2=0x7fd67407ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.53:6824/2055146181,v1:172.21.15.53:6825/2055146181] conn(0x7fd67407aa40 0x7fd67407ced0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6740744b0 msgr2=0x7fd6740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd6740744b0 0x7fd6740769c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd68c006fd0 tx=0x7fd68c008040 comp rx=0 tx=0).stop 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 msgr2=0x7fd6940b6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 --2- 172.21.15.28:0/781810592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6940d5690 0x7fd6940b6790 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fd68400cd80 tx=0x7fd68400a650 comp rx=0 tx=0).stop 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd6937fe640 1 -- 172.21.15.28:0/781810592 reap_dead start 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 shutdown_connections 2024-04-25T19:13:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 >> 172.21.15.28:0/781810592 conn(0x7fd6940d2c60 msgr2=0x7fd6941628c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:41.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 shutdown_connections 2024-04-25T19:13:41.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.317+0000 7fd69993f640 1 -- 172.21.15.28:0/781810592 wait complete. 2024-04-25T19:13:41.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T19:13:41.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.440+0000 7f92a790d640 1 Processor -- start 2024-04-25T19:13:41.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.441+0000 7f92a790d640 1 -- start start 2024-04-25T19:13:41.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a790d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 0x7f92a00b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a790d640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f92a00b81a0 0x7f92a00d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a790d640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92a00d2ee0 con 0x7f92a00d2ad0 2024-04-25T19:13:41.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a790d640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f92a00b4f50 con 0x7f92a00b81a0 2024-04-25T19:13:41.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a790d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f92a0164b20 con 0x7f92a0160250 2024-04-25T19:13:41.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a5eaa640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f92a00d2ad0 0x7f92a00d0830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38908/0 (socket says 172.21.15.28:38908) 2024-04-25T19:13:41.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a5eaa640 1 -- 172.21.15.28:0/3521502880 learned_addr learned my addr 172.21.15.28:0/3521502880 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a4ea8640 1 --2- 172.21.15.28:0/3521502880 >> v2:172.21.15.53:3300/0 conn(0x7f92a00b81a0 0x7f92a00d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f92a56a9640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 0x7f92a00b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1837165430 0 0) 0x7f9298001110 con 0x7f92a00d2ad0 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.442+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9288003520 con 0x7f92a00d2ad0 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4022177109 0 0) 0x7f9288003520 con 0x7f92a00d2ad0 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 msgr2=0x7f92a00b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 0x7f92a00b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 >> v2:172.21.15.53:3300/0 conn(0x7f92a00b81a0 msgr2=0x7f92a00d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 --2- 172.21.15.28:0/3521502880 >> v2:172.21.15.53:3300/0 conn(0x7f92a00b81a0 0x7f92a00d2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a0163d60 con 0x7f92a00d2ad0 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f9298001840 con 0x7f92a00d2ad0 2024-04-25T19:13:41.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 >> v1:172.21.15.28:6789/0 conn(0x7f92a00d2ad0 legacy=0x7f92a00d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f9288003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9288004800 0x7f9288004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 0x7f9288005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92a00b4f50 con 0x7f92880039b0 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f92a0164b20 con 0x7f9288004800 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f928800ba70 con 0x7f92a0160250 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a56a9640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f9288003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a4ea8640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9288004800 0x7f9288004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a5eaa640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 0x7f9288005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a56a9640 1 -- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 msgr2=0x7f9288005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a56a9640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 0x7f9288005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a56a9640 1 -- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9288004800 msgr2=0x7f9288004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a56a9640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9288004800 0x7f9288004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a56a9640 1 -- 172.21.15.28:0/3521502880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9288003520 con 0x7f92880039b0 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.443+0000 7f92a5eaa640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a0160250 0x7f9288005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.444+0000 7f92a56a9640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f9288003e20 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9290009f90 tx=0x7f9288005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53d86de3fd05373 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.444+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290009c10 con 0x7f92880039b0 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.444+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9290036070 con 0x7f92880039b0 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.444+0000 7f929ffff640 1 -- 172.21.15.28:0/3521502880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290031810 con 0x7f92880039b0 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.445+0000 7f92a790d640 1 -- 172.21.15.28:0/3521502880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 msgr2=0x7f9288003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.445+0000 7f92a790d640 1 --2- 172.21.15.28:0/3521502880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f9288003e20 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9290009f90 tx=0x7f9288005830 comp rx=0 tx=0).stop 2024-04-25T19:13:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.445+0000 7f92a5eaa640 1 -- 172.21.15.28:0/3521502880 reap_dead start 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.445+0000 7f92a790d640 1 -- 172.21.15.28:0/3521502880 shutdown_connections 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.445+0000 7f92a790d640 1 -- 172.21.15.28:0/3521502880 >> 172.21.15.28:0/3521502880 conn(0x7f92a006ee30 msgr2=0x7f92a006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.445+0000 7f92a790d640 1 -- 172.21.15.28:0/3521502880 shutdown_connections 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.445+0000 7f92a790d640 1 -- 172.21.15.28:0/3521502880 wait complete. 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.447+0000 7f92a790d640 1 Processor -- start 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.447+0000 7f92a790d640 1 -- start start 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a790d640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f92a01f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a790d640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a00ca310 0x7f92a01f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a790d640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92a00b5da0 0x7f92a00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a790d640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92a0164600 con 0x7f92880039b0 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a790d640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f92a0164320 con 0x7f92a00b5da0 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a790d640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f92a00d54c0 con 0x7f92a00ca310 2024-04-25T19:13:41.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a56a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f92a01f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a56a9640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f92a01f7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58552/0 (socket says 172.21.15.28:58552) 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a56a9640 1 -- 172.21.15.28:0/2594793450 learned_addr learned my addr 172.21.15.28:0/2594793450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a5eaa640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92a00b5da0 0x7f92a00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a4ea8640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a00ca310 0x7f92a01f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a56a9640 1 -- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a00ca310 msgr2=0x7f92a01f7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a56a9640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a00ca310 0x7f92a01f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a56a9640 1 -- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92a00b5da0 msgr2=0x7f92a00b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a56a9640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92a00b5da0 0x7f92a00b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a56a9640 1 -- 172.21.15.28:0/2594793450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a00b66f0 con 0x7f92880039b0 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.448+0000 7f92a5eaa640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92a00b5da0 0x7f92a00b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.449+0000 7f92a56a9640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f92a01f7a20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f929000e230 tx=0x7f9290009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.449+0000 7f929dffb640 1 -- 172.21.15.28:0/2594793450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290009c10 con 0x7f92880039b0 2024-04-25T19:13:41.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.449+0000 7f929dffb640 1 -- 172.21.15.28:0/2594793450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9290045630 con 0x7f92880039b0 2024-04-25T19:13:41.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.449+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a00cab20 con 0x7f92880039b0 2024-04-25T19:13:41.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.449+0000 7f929dffb640 1 -- 172.21.15.28:0/2594793450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9290045a20 con 0x7f92880039b0 2024-04-25T19:13:41.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.449+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a00cb0e0 con 0x7f92880039b0 2024-04-25T19:13:41.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.450+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a0050740 con 0x7f92880039b0 2024-04-25T19:13:41.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.451+0000 7f929dffb640 1 -- 172.21.15.28:0/2594793450 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9290005d10 con 0x7f92880039b0 2024-04-25T19:13:41.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.451+0000 7f929dffb640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f928c0744e0 0x7f928c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.451+0000 7f929dffb640 1 -- 172.21.15.28:0/2594793450 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f92900b6870 con 0x7f92880039b0 2024-04-25T19:13:41.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.451+0000 7f92a4ea8640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f928c0744e0 0x7f928c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.452+0000 7f92a4ea8640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f928c0744e0 0x7f928c0769f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f92940090e0 tx=0x7f9294008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.456+0000 7f929dffb640 1 -- 172.21.15.28:0/2594793450 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92a00b66f0 con 0x7f92880039b0 2024-04-25T19:13:41.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.642+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f92a00cb320 con 0x7f928c0744e0 2024-04-25T19:13:41.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.642+0000 7f929dffb640 1 -- 172.21.15.28:0/2594793450 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9294004020 con 0x7f928c0744e0 2024-04-25T19:13:41.644 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T19:13:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f928c0744e0 msgr2=0x7f928c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f928c0744e0 0x7f928c0769f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f92940090e0 tx=0x7f9294008040 comp rx=0 tx=0).stop 2024-04-25T19:13:41.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 msgr2=0x7f92a01f7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f92a01f7a20 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f929000e230 tx=0x7f9290009750 comp rx=0 tx=0).stop 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 shutdown_connections 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f928c0744e0 0x7f928c0769f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92a00b5da0 0x7f92a00b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f92a00ca310 0x7f92a01f7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 --2- 172.21.15.28:0/2594793450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92880039b0 0x7f92a01f7a20 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 >> 172.21.15.28:0/2594793450 conn(0x7f92a006ee30 msgr2=0x7f92a00d5750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 shutdown_connections 2024-04-25T19:13:41.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.644+0000 7f92a790d640 1 -- 172.21.15.28:0/2594793450 wait complete. 2024-04-25T19:13:41.657 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T19:13:41.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T19:13:41.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.771+0000 7f72165f3640 1 Processor -- start 2024-04-25T19:13:41.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.771+0000 7f72165f3640 1 -- start start 2024-04-25T19:13:41.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f72165f3640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f72100b47d0 0x7f72101601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f72165f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f721006e5e0 0x7f72100b9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f72165f3640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72100d12e0 con 0x7f72100d5690 2024-04-25T19:13:41.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f72165f3640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f72100d15c0 con 0x7f72100b47d0 2024-04-25T19:13:41.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f72165f3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f72100d4bb0 con 0x7f721006e5e0 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f720f7fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f72100d5690 0x7f721006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38922/0 (socket says 172.21.15.28:38922) 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f720f7fe640 1 -- 172.21.15.28:0/1137975502 learned_addr learned my addr 172.21.15.28:0/1137975502 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f720ffff640 1 --2- 172.21.15.28:0/1137975502 >> v2:172.21.15.53:3300/0 conn(0x7f72100b47d0 0x7f72101601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f7214b90640 1 --2- 172.21.15.28:0/1137975502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f721006e5e0 0x7f72100b9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f720effd640 1 -- 172.21.15.28:0/1137975502 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 536214106 0 0) 0x7f72100d12e0 con 0x7f72100d5690 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f720effd640 1 -- 172.21.15.28:0/1137975502 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71fc003520 con 0x7f72100d5690 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f7214b90640 1 -- 172.21.15.28:0/1137975502 >> v2:172.21.15.53:3300/0 conn(0x7f72100b47d0 msgr2=0x7f72101601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f7214b90640 1 --2- 172.21.15.28:0/1137975502 >> v2:172.21.15.53:3300/0 conn(0x7f72100b47d0 0x7f72101601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f7214b90640 1 -- 172.21.15.28:0/1137975502 >> v1:172.21.15.28:6789/0 conn(0x7f72100d5690 legacy=0x7f721006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.773+0000 7f7214b90640 1 -- 172.21.15.28:0/1137975502 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72100d4e90 con 0x7f721006e5e0 2024-04-25T19:13:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.774+0000 7f7214b90640 1 --2- 172.21.15.28:0/1137975502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f721006e5e0 0x7f72100b9200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7208009d90 tx=0x7f720802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cedf7e7a2a5c064 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.774+0000 7f720effd640 1 -- 172.21.15.28:0/1137975502 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208007740 con 0x7f721006e5e0 2024-04-25T19:13:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.774+0000 7f720effd640 1 -- 172.21.15.28:0/1137975502 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72080078e0 con 0x7f721006e5e0 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.774+0000 7f720effd640 1 -- 172.21.15.28:0/1137975502 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7208007c90 con 0x7f721006e5e0 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.775+0000 7f72165f3640 1 -- 172.21.15.28:0/1137975502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f721006e5e0 msgr2=0x7f72100b9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.775+0000 7f72165f3640 1 --2- 172.21.15.28:0/1137975502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f721006e5e0 0x7f72100b9200 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7208009d90 tx=0x7f720802a660 comp rx=0 tx=0).stop 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.775+0000 7f72165f3640 1 -- 172.21.15.28:0/1137975502 shutdown_connections 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.775+0000 7f72165f3640 1 --2- 172.21.15.28:0/1137975502 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f721006e5e0 0x7f72100b9200 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.775+0000 7f72165f3640 1 --2- 172.21.15.28:0/1137975502 >> v2:172.21.15.53:3300/0 conn(0x7f72100b47d0 0x7f72101601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.775+0000 7f72165f3640 1 -- 172.21.15.28:0/1137975502 >> 172.21.15.28:0/1137975502 conn(0x7f72100d2c60 msgr2=0x7f72100d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.775+0000 7f72165f3640 1 -- 172.21.15.28:0/1137975502 shutdown_connections 2024-04-25T19:13:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.775+0000 7f72165f3640 1 -- 172.21.15.28:0/1137975502 wait complete. 2024-04-25T19:13:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.777+0000 7f72165f3640 1 Processor -- start 2024-04-25T19:13:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.777+0000 7f72165f3640 1 -- start start 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f72165f3640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72100b47d0 0x7f7210151090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f72165f3640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72100d5690 0x7f7210163390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f72165f3640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72100b7680 0x7f72100b7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f72165f3640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72100b4d00 con 0x7f72100d5690 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f72165f3640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7210160730 con 0x7f72100b7680 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f72165f3640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f72100b4e80 con 0x7f72100b47d0 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f720f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72100d5690 0x7f7210163390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f720f7fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72100d5690 0x7f7210163390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58554/0 (socket says 172.21.15.28:58554) 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.778+0000 7f720f7fe640 1 -- 172.21.15.28:0/1083802460 learned_addr learned my addr 172.21.15.28:0/1083802460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720ffff640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72100b47d0 0x7f7210151090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f7214b90640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72100b7680 0x7f72100b7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720f7fe640 1 -- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72100b47d0 msgr2=0x7f7210151090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720f7fe640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f72100b47d0 0x7f7210151090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720f7fe640 1 -- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72100b7680 msgr2=0x7f72100b7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720f7fe640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72100b7680 0x7f72100b7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720f7fe640 1 -- 172.21.15.28:0/1083802460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72100b7fd0 con 0x7f72100d5690 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720f7fe640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72100d5690 0x7f7210163390 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f72000010e0 tx=0x7f720000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200017070 con 0x7f72100d5690 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7200005d00 con 0x7f72100d5690 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72100cad20 con 0x7f72100d5690 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7200007310 con 0x7f72100d5690 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.779+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72100c9f60 con 0x7f72100d5690 2024-04-25T19:13:41.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.780+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f721006a100 con 0x7f72100d5690 2024-04-25T19:13:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.781+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f720001a070 con 0x7f72100d5690 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.781+0000 7f720cff9640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71f00744e0 0x7f71f00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.782+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f72000944e0 con 0x7f72100d5690 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.782+0000 7f720ffff640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71f00744e0 0x7f71f00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.782+0000 7f720cff9640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f71f007aa70 0x7f71f007cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.782+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71f007d5d0 con 0x7f71f007aa70 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.782+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f720009b050 con 0x7f72100d5690 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.782+0000 7f7214b90640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f71f007aa70 0x7f71f007cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.782+0000 7f720ffff640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71f00744e0 0x7f71f00769f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f72100d0650 tx=0x7f71f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.782+0000 7f7214b90640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f71f007aa70 0x7f71f007cf00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.783+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 <== osd.5 v2:172.21.15.53:6800/2342659583 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f71f007d5d0 con 0x7f71f007aa70 2024-04-25T19:13:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.796+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 --> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f721015a140 con 0x7f71f007aa70 2024-04-25T19:13:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.797+0000 7f720cff9640 1 -- 172.21.15.28:0/1083802460 <== osd.5 v2:172.21.15.53:6800/2342659583 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f721015a140 con 0x7f71f007aa70 2024-04-25T19:13:41.799 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:41.799 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T19:13:41.799 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.797+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f71f007aa70 msgr2=0x7f71f007cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.53:6800/2342659583,v1:172.21.15.53:6801/2342659583] conn(0x7f71f007aa70 0x7f71f007cf00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71f00744e0 msgr2=0x7f71f00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71f00744e0 0x7f71f00769f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f72100d0650 tx=0x7f71f8008040 comp rx=0 tx=0).stop 2024-04-25T19:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72100d5690 msgr2=0x7f7210163390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 --2- 172.21.15.28:0/1083802460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72100d5690 0x7f7210163390 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f72000010e0 tx=0x7f720000c890 comp rx=0 tx=0).stop 2024-04-25T19:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f7214b90640 1 -- 172.21.15.28:0/1083802460 reap_dead start 2024-04-25T19:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 shutdown_connections 2024-04-25T19:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 >> 172.21.15.28:0/1083802460 conn(0x7f72100d2c60 msgr2=0x7f721015d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 shutdown_connections 2024-04-25T19:13:41.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.798+0000 7f72165f3640 1 -- 172.21.15.28:0/1083802460 wait complete. 2024-04-25T19:13:41.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T19:13:41.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.921+0000 7f3bd0157640 1 Processor -- start 2024-04-25T19:13:41.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.921+0000 7f3bd0157640 1 -- start start 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bd0157640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3bc8164500 0x7f3bc815a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bd0157640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc806d0b0 con 0x7f3bc809b540 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bd0157640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bc80b2610 con 0x7f3bc806db50 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bd0157640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f3bc806d6c0 con 0x7f3bc8164500 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bce6f4640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3bc809b540 0x7f3bc81619c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38926/0 (socket says 172.21.15.28:38926) 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bce6f4640 1 -- 172.21.15.28:0/2500922798 learned_addr learned my addr 172.21.15.28:0/2500922798 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bcd6f2640 1 --2- 172.21.15.28:0/2500922798 >> v2:172.21.15.53:3300/0 conn(0x7f3bc8164500 0x7f3bc815a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1528860950 0 0) 0x7f3bc806d0b0 con 0x7f3bc809b540 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bb4003520 con 0x7f3bc809b540 2024-04-25T19:13:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 405157464 0 0) 0x7f3bb4003520 con 0x7f3bc809b540 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 >> v2:172.21.15.53:3300/0 conn(0x7f3bc8164500 msgr2=0x7f3bc815a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bccef1640 1 --2- 172.21.15.28:0/2500922798 >> v2:172.21.15.53:3300/0 conn(0x7f3bc8164500 0x7f3bc815a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 >> v1:172.21.15.53:6789/0 conn(0x7f3bc806db50 legacy=0x7f3bc8163df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc806d350 con 0x7f3bc809b540 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f3bc0001a10 con 0x7f3bc809b540 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.923+0000 7f3bcd6f2640 1 --2- 172.21.15.28:0/2500922798 >> v2:172.21.15.53:3300/0 conn(0x7f3bc8164500 0x7f3bc815a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 >> v1:172.21.15.28:6789/0 conn(0x7f3bc809b540 legacy=0x7f3bc81619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bccef1640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb40039b0 0x7f3bb4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bccef1640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb4004800 0x7f3bb4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bccef1640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bb4005250 0x7f3bb400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc806d6c0 con 0x7f3bb4004800 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3bb400c1e0 con 0x7f3bb40039b0 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f3bb400c3e0 con 0x7f3bb4005250 2024-04-25T19:13:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bcd6f2640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb4004800 0x7f3bb4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bcdef3640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb40039b0 0x7f3bb4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bce6f4640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bb4005250 0x7f3bb400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bcd6f2640 1 -- 172.21.15.28:0/2500922798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bb4005250 msgr2=0x7f3bb400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bcd6f2640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bb4005250 0x7f3bb400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bcd6f2640 1 -- 172.21.15.28:0/2500922798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb40039b0 msgr2=0x7f3bb4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bcd6f2640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb40039b0 0x7f3bb4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bcd6f2640 1 -- 172.21.15.28:0/2500922798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bb4003520 con 0x7f3bb4004800 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.924+0000 7f3bce6f4640 1 -- 172.21.15.28:0/2500922798 reap_dead start 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.925+0000 7f3bcd6f2640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb4004800 0x7f3bb4004c10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009c90 tx=0x7f3bb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400681c5d64b1d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.925+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8036070 con 0x7f3bb4004800 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.925+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bb8031400 con 0x7f3bb4004800 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.925+0000 7f3bccef1640 1 -- 172.21.15.28:0/2500922798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8009860 con 0x7f3bb4004800 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.926+0000 7f3bd0157640 1 -- 172.21.15.28:0/2500922798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb4004800 msgr2=0x7f3bb4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.926+0000 7f3bd0157640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb4004800 0x7f3bb4004c10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009c90 tx=0x7f3bb4005830 comp rx=0 tx=0).stop 2024-04-25T19:13:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.926+0000 7f3bd0157640 1 -- 172.21.15.28:0/2500922798 shutdown_connections 2024-04-25T19:13:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.926+0000 7f3bd0157640 1 --2- 172.21.15.28:0/2500922798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb4004800 0x7f3bb4004c10 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.926+0000 7f3bd0157640 1 -- 172.21.15.28:0/2500922798 >> 172.21.15.28:0/2500922798 conn(0x7f3bc8073950 msgr2=0x7f3bc80b70f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.926+0000 7f3bd0157640 1 -- 172.21.15.28:0/2500922798 shutdown_connections 2024-04-25T19:13:41.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.926+0000 7f3bd0157640 1 -- 172.21.15.28:0/2500922798 wait complete. 2024-04-25T19:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.927+0000 7f3bd0157640 1 Processor -- start 2024-04-25T19:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.928+0000 7f3bd0157640 1 -- start start 2024-04-25T19:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bd0157640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb4004800 0x7f3bc8156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bd0157640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bc809e620 0x7f3bc809ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bd0157640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc8160670 0x7f3bc8160a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bd0157640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc8164ae0 con 0x7f3bc8160670 2024-04-25T19:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bd0157640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3bc80b28f0 con 0x7f3bb4004800 2024-04-25T19:13:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bd0157640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f3bc8164c60 con 0x7f3bc809e620 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc8160670 0x7f3bc8160a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc8160670 0x7f3bc8160a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58578/0 (socket says 172.21.15.28:58578) 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 -- 172.21.15.28:0/12966626 learned_addr learned my addr 172.21.15.28:0/12966626 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bcdef3640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb4004800 0x7f3bc8156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bcd6f2640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bc809e620 0x7f3bc809ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 -- 172.21.15.28:0/12966626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bc809e620 msgr2=0x7f3bc809ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bc809e620 0x7f3bc809ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 -- 172.21.15.28:0/12966626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb4004800 msgr2=0x7f3bc8156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb4004800 0x7f3bc8156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 -- 172.21.15.28:0/12966626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bc8160fc0 con 0x7f3bc8160670 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.929+0000 7f3bce6f4640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc8160670 0x7f3bc8160a80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8072110 tx=0x7f3bc0007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.930+0000 7f3bbe7fc640 1 -- 172.21.15.28:0/12966626 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc000abc0 con 0x7f3bc8160670 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.930+0000 7f3bbe7fc640 1 -- 172.21.15.28:0/12966626 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc0018070 con 0x7f3bc8160670 2024-04-25T19:13:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.930+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bc809cd10 con 0x7f3bc8160670 2024-04-25T19:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.930+0000 7f3bbe7fc640 1 -- 172.21.15.28:0/12966626 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc00133f0 con 0x7f3bc8160670 2024-04-25T19:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.930+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bc809d2d0 con 0x7f3bc8160670 2024-04-25T19:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.931+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bc809f050 con 0x7f3bc8160670 2024-04-25T19:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.931+0000 7f3bbe7fc640 1 -- 172.21.15.28:0/12966626 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3bc0013590 con 0x7f3bc8160670 2024-04-25T19:13:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.932+0000 7f3bbe7fc640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3bac0744b0 0x7f3bac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.932+0000 7f3bbe7fc640 1 -- 172.21.15.28:0/12966626 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f3bc00956c0 con 0x7f3bc8160670 2024-04-25T19:13:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.932+0000 7f3bcdef3640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3bac0744b0 0x7f3bac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.932+0000 7f3bcdef3640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3bac0744b0 0x7f3bac0769c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80d2be0 tx=0x7f3bc4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:41.937+0000 7f3bbe7fc640 1 -- 172.21.15.28:0/12966626 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3bc8160fc0 con 0x7f3bc8160670 2024-04-25T19:13:42.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.123+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f3bc8075b10 con 0x7f3bac0744b0 2024-04-25T19:13:42.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.123+0000 7f3bbe7fc640 1 -- 172.21.15.28:0/12966626 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3bc8075b10 con 0x7f3bac0744b0 2024-04-25T19:13:42.125 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T19:13:42.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3bac0744b0 msgr2=0x7f3bac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3bac0744b0 0x7f3bac0769c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80d2be0 tx=0x7f3bc4009040 comp rx=0 tx=0).stop 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc8160670 msgr2=0x7f3bc8160a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc8160670 0x7f3bc8160a80 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8072110 tx=0x7f3bc0007b20 comp rx=0 tx=0).stop 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 shutdown_connections 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f3bac0744b0 0x7f3bac0769c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bc8160670 0x7f3bc8160a80 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3bc809e620 0x7f3bc809ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 --2- 172.21.15.28:0/12966626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bb4004800 0x7f3bc8156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 >> 172.21.15.28:0/12966626 conn(0x7f3bc8073950 msgr2=0x7f3bc8070210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 shutdown_connections 2024-04-25T19:13:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.125+0000 7f3bd0157640 1 -- 172.21.15.28:0/12966626 wait complete. 2024-04-25T19:13:42.138 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T19:13:42.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T19:13:42.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.252+0000 7f2fcba0c640 1 Processor -- start 2024-04-25T19:13:42.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.252+0000 7f2fcba0c640 1 -- start start 2024-04-25T19:13:42.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.253+0000 7f2fcba0c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 0x7f2fc406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.253+0000 7f2fcba0c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fc40d12e0 con 0x7f2fc40b47d0 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.253+0000 7f2fcba0c640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fc40d15c0 con 0x7f2fc40b4d00 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.253+0000 7f2fcba0c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc40d4bb0 con 0x7f2fc40d5690 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.253+0000 7f2fc97a8640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2fc40b47d0 0x7f2fc41601f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38932/0 (socket says 172.21.15.28:38932) 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.253+0000 7f2fc97a8640 1 -- 172.21.15.28:0/366657406 learned_addr learned my addr 172.21.15.28:0/366657406 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 179458169 0 0) 0x7f2fc40d12e0 con 0x7f2fc40b47d0 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fc8fa7640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 0x7f2fc406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fb0003520 con 0x7f2fc40b47d0 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1110552153 0 0) 0x7f2fb0003520 con 0x7f2fc40b47d0 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 msgr2=0x7f2fc406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 0x7f2fc406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 >> v1:172.21.15.53:6789/0 conn(0x7f2fc40b4d00 legacy=0x7f2fc40b9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc40d4e90 con 0x7f2fc40b47d0 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f2fc0001690 con 0x7f2fc40b47d0 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 >> v1:172.21.15.28:6789/0 conn(0x7f2fc40b47d0 legacy=0x7f2fc41601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb00039e0 0x7f2fb0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fb0004800 0x7f2fb0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 0x7f2fb0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40d4bb0 con 0x7f2fb00039e0 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb000bb50 con 0x7f2fb0004800 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2fb000bd00 con 0x7f2fc40d5690 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fc97a8640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb00039e0 0x7f2fb0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fc8fa7640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fb0004800 0x7f2fb0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.254+0000 7f2fc9fa9640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 0x7f2fb0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fc97a8640 1 -- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 msgr2=0x7f2fb0005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fc97a8640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 0x7f2fb0005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fc97a8640 1 -- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fb0004800 msgr2=0x7f2fb0004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fc97a8640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fb0004800 0x7f2fb0004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fc97a8640 1 -- 172.21.15.28:0/366657406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb0003520 con 0x7f2fb00039e0 2024-04-25T19:13:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fc9fa9640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fc40d5690 0x7f2fb0005220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fc97a8640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb00039e0 0x7f2fb0003e50 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40d0650 tx=0x7f2fb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffefb36e241f421b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0007d10 con 0x7f2fb00039e0 2024-04-25T19:13:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.255+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fc0005ad0 con 0x7f2fb00039e0 2024-04-25T19:13:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.256+0000 7f2fbbfff640 1 -- 172.21.15.28:0/366657406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0007070 con 0x7f2fb00039e0 2024-04-25T19:13:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.256+0000 7f2fcba0c640 1 -- 172.21.15.28:0/366657406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb00039e0 msgr2=0x7f2fb0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.256+0000 7f2fcba0c640 1 --2- 172.21.15.28:0/366657406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fb00039e0 0x7f2fb0003e50 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40d0650 tx=0x7f2fb0005830 comp rx=0 tx=0).stop 2024-04-25T19:13:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.256+0000 7f2fc9fa9640 1 -- 172.21.15.28:0/366657406 reap_dead start 2024-04-25T19:13:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.256+0000 7f2fcba0c640 1 -- 172.21.15.28:0/366657406 shutdown_connections 2024-04-25T19:13:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.256+0000 7f2fcba0c640 1 -- 172.21.15.28:0/366657406 >> 172.21.15.28:0/366657406 conn(0x7f2fc40d2c60 msgr2=0x7f2fc40d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.256+0000 7f2fcba0c640 1 -- 172.21.15.28:0/366657406 shutdown_connections 2024-04-25T19:13:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.256+0000 7f2fcba0c640 1 -- 172.21.15.28:0/366657406 wait complete. 2024-04-25T19:13:42.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.258+0000 7f2fcba0c640 1 Processor -- start 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.258+0000 7f2fcba0c640 1 -- start start 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.259+0000 7f2fcba0c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fb00039e0 0x7f2fc41e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.259+0000 7f2fcba0c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc41e8390 0x7f2fc4158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.259+0000 7f2fcba0c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fc40b5880 0x7f2fc40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.259+0000 7f2fcba0c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40b9a90 con 0x7f2fc41e8390 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.259+0000 7f2fcba0c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc40b9910 con 0x7f2fc40b5880 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.259+0000 7f2fcba0c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f2fc40b9c10 con 0x7f2fb00039e0 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.259+0000 7f2fc8fa7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc41e8390 0x7f2fc4158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.259+0000 7f2fc8fa7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc41e8390 0x7f2fc4158270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58602/0 (socket says 172.21.15.28:58602) 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc8fa7640 1 -- 172.21.15.28:0/3902730695 learned_addr learned my addr 172.21.15.28:0/3902730695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc9fa9640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fc40b5880 0x7f2fc40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc97a8640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fb00039e0 0x7f2fc41e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc8fa7640 1 -- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fb00039e0 msgr2=0x7f2fc41e5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc8fa7640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fb00039e0 0x7f2fc41e5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc8fa7640 1 -- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fc40b5880 msgr2=0x7f2fc40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc8fa7640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fc40b5880 0x7f2fc40b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc8fa7640 1 -- 172.21.15.28:0/3902730695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc40b61d0 con 0x7f2fc41e8390 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc9fa9640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2fc40b5880 0x7f2fc40b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc97a8640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f2fb00039e0 0x7f2fc41e5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fc8fa7640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc41e8390 0x7f2fc4158270 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40119d0 tx=0x7f2fb4011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb400fb50 con 0x7f2fc41e8390 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb400fcf0 con 0x7f2fc41e8390 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc41e8bc0 con 0x7f2fc41e8390 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb400fea0 con 0x7f2fc41e8390 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.260+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc415a0f0 con 0x7f2fc41e8390 2024-04-25T19:13:42.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.261+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2fc41e8ea0 con 0x7f2fc41e8390 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.262+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2fb4012040 con 0x7f2fc41e8390 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.263+0000 7f2fb9ffb640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2fa80744b0 0x7f2fa80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.263+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f2fb4095f10 con 0x7f2fc41e8390 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.263+0000 7f2fc97a8640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2fa80744b0 0x7f2fa80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.263+0000 7f2fb9ffb640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f2fa807aa40 0x7f2fa807ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.263+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2fb400de90 con 0x7f2fa807aa40 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.263+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f2fb40962c0 con 0x7f2fc41e8390 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.263+0000 7f2fc97a8640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2fa80744b0 0x7f2fa80769c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0032000 tx=0x7f2fc0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.263+0000 7f2fc9fa9640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f2fa807aa40 0x7f2fa807ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.264+0000 7f2fc9fa9640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f2fa807aa40 0x7f2fa807ced0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.265+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 <== osd.6 v2:172.21.15.53:6809/3653668189 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2fb400de90 con 0x7f2fa807aa40 2024-04-25T19:13:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.278+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 --> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2fc41e8970 con 0x7f2fa807aa40 2024-04-25T19:13:42.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fb9ffb640 1 -- 172.21.15.28:0/3902730695 <== osd.6 v2:172.21.15.53:6809/3653668189 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2fc41e8970 con 0x7f2fa807aa40 2024-04-25T19:13:42.281 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:42.281 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T19:13:42.281 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f2fa807aa40 msgr2=0x7f2fa807ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.53:6809/3653668189,v1:172.21.15.53:6811/3653668189] conn(0x7f2fa807aa40 0x7f2fa807ced0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2fa80744b0 msgr2=0x7f2fa80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f2fa80744b0 0x7f2fa80769c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0032000 tx=0x7f2fc0009030 comp rx=0 tx=0).stop 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc41e8390 msgr2=0x7f2fc4158270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 --2- 172.21.15.28:0/3902730695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fc41e8390 0x7f2fc4158270 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f2fb40119d0 tx=0x7f2fb4011e90 comp rx=0 tx=0).stop 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fc9fa9640 1 -- 172.21.15.28:0/3902730695 reap_dead start 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 shutdown_connections 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 >> 172.21.15.28:0/3902730695 conn(0x7f2fc40d2c60 msgr2=0x7f2fc40b9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 shutdown_connections 2024-04-25T19:13:42.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.279+0000 7f2fcba0c640 1 -- 172.21.15.28:0/3902730695 wait complete. 2024-04-25T19:13:42.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T19:13:42.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.402+0000 7f5c9e385640 1 Processor -- start 2024-04-25T19:13:42.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.402+0000 7f5c9e385640 1 -- start start 2024-04-25T19:13:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c9e385640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c980d2d10 0x7f5c980d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c9e385640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c980b92a0 con 0x7f5c981686b0 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c9e385640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9809cea0 con 0x7f5c980b3d00 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c9e385640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f5c9806e690 con 0x7f5c980d2d10 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c9c922640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5c981686b0 0x7f5c9816aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38938/0 (socket says 172.21.15.28:38938) 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c9c922640 1 -- 172.21.15.28:0/4113037033 learned_addr learned my addr 172.21.15.28:0/4113037033 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c97fff640 1 --2- 172.21.15.28:0/4113037033 >> v2:172.21.15.53:3300/0 conn(0x7f5c980d2d10 0x7f5c980d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216703252 0 0) 0x7f5c980b92a0 con 0x7f5c981686b0 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c84003520 con 0x7f5c981686b0 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 804368657 0 0) 0x7f5c9809cea0 con 0x7f5c980b3d00 2024-04-25T19:13:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c980b92a0 con 0x7f5c980b3d00 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 256832245 0 0) 0x7f5c84003520 con 0x7f5c981686b0 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 >> v2:172.21.15.53:3300/0 conn(0x7f5c980d2d10 msgr2=0x7f5c980d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 --2- 172.21.15.28:0/4113037033 >> v2:172.21.15.53:3300/0 conn(0x7f5c980d2d10 0x7f5c980d21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 >> v1:172.21.15.53:6789/0 conn(0x7f5c980b3d00 legacy=0x7f5c980b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9806e8e0 con 0x7f5c981686b0 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f5c900019f0 con 0x7f5c981686b0 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.404+0000 7f5c97fff640 1 --2- 172.21.15.28:0/4113037033 >> v2:172.21.15.53:3300/0 conn(0x7f5c980d2d10 0x7f5c980d21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 >> v1:172.21.15.28:6789/0 conn(0x7f5c981686b0 legacy=0x7f5c9816aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c96ffd640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84003b90 0x7f5c84003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c96ffd640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84004800 0x7f5c84004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c96ffd640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c84005280 0x7f5c8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9806e690 con 0x7f5c84004800 2024-04-25T19:13:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5c8400c230 con 0x7f5c84003b90 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5c8400c430 con 0x7f5c84005280 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c977fe640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84004800 0x7f5c84004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c97fff640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84003b90 0x7f5c84003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c9c922640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c84005280 0x7f5c8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c977fe640 1 -- 172.21.15.28:0/4113037033 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c84005280 msgr2=0x7f5c8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c977fe640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c84005280 0x7f5c8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c977fe640 1 -- 172.21.15.28:0/4113037033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84003b90 msgr2=0x7f5c84003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c977fe640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84003b90 0x7f5c84003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c977fe640 1 -- 172.21.15.28:0/4113037033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c84003520 con 0x7f5c84004800 2024-04-25T19:13:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.405+0000 7f5c9c922640 1 -- 172.21.15.28:0/4113037033 reap_dead start 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.406+0000 7f5c977fe640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84004800 0x7f5c84004c10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5c980d0420 tx=0x7f5c84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac984203cdeeb27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.406+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c880014f0 con 0x7f5c84004800 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.406+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c880052d0 con 0x7f5c84004800 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.406+0000 7f5c96ffd640 1 -- 172.21.15.28:0/4113037033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8802abe0 con 0x7f5c84004800 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.407+0000 7f5c9e385640 1 -- 172.21.15.28:0/4113037033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84004800 msgr2=0x7f5c84004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.407+0000 7f5c9e385640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84004800 0x7f5c84004c10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5c980d0420 tx=0x7f5c84005830 comp rx=0 tx=0).stop 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.407+0000 7f5c9e385640 1 -- 172.21.15.28:0/4113037033 shutdown_connections 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.407+0000 7f5c9e385640 1 --2- 172.21.15.28:0/4113037033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84004800 0x7f5c84004c10 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.407+0000 7f5c9e385640 1 -- 172.21.15.28:0/4113037033 >> 172.21.15.28:0/4113037033 conn(0x7f5c980d0450 msgr2=0x7f5c980d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.407+0000 7f5c9e385640 1 -- 172.21.15.28:0/4113037033 shutdown_connections 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.407+0000 7f5c9e385640 1 -- 172.21.15.28:0/4113037033 wait complete. 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.408+0000 7f5c9e385640 1 Processor -- start 2024-04-25T19:13:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.409+0000 7f5c9e385640 1 -- start start 2024-04-25T19:13:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9e385640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84004800 0x7f5c9814ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9e385640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c980c8d90 0x7f5c980c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9e385640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c980b86b0 0x7f5c980b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9e385640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c980d3120 con 0x7f5c980b86b0 2024-04-25T19:13:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9e385640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f5c9806ebc0 con 0x7f5c84004800 2024-04-25T19:13:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9e385640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f5c980d3600 con 0x7f5c980c8d90 2024-04-25T19:13:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9c922640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c980b86b0 0x7f5c980b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9c922640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c980b86b0 0x7f5c980b1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58620/0 (socket says 172.21.15.28:58620) 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9c922640 1 -- 172.21.15.28:0/3168023572 learned_addr learned my addr 172.21.15.28:0/3168023572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c977fe640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c980c8d90 0x7f5c980c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c97fff640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84004800 0x7f5c9814ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9c922640 1 -- 172.21.15.28:0/3168023572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c980c8d90 msgr2=0x7f5c980c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9c922640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c980c8d90 0x7f5c980c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9c922640 1 -- 172.21.15.28:0/3168023572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84004800 msgr2=0x7f5c9814ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9c922640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84004800 0x7f5c9814ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.410+0000 7f5c9c922640 1 -- 172.21.15.28:0/3168023572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c980c7cf0 con 0x7f5c980b86b0 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.411+0000 7f5c9c922640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c980b86b0 0x7f5c980b1bf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5c980d19d0 tx=0x7f5c90007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.411+0000 7f5c94ff9640 1 -- 172.21.15.28:0/3168023572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c9000aa90 con 0x7f5c980b86b0 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.411+0000 7f5c94ff9640 1 -- 172.21.15.28:0/3168023572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c90018070 con 0x7f5c980b86b0 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.411+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c980c7fd0 con 0x7f5c980b86b0 2024-04-25T19:13:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.411+0000 7f5c94ff9640 1 -- 172.21.15.28:0/3168023572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c900133f0 con 0x7f5c980b86b0 2024-04-25T19:13:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.411+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c980c8590 con 0x7f5c980b86b0 2024-04-25T19:13:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.412+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c9806f450 con 0x7f5c980b86b0 2024-04-25T19:13:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.413+0000 7f5c94ff9640 1 -- 172.21.15.28:0/3168023572 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5c90013590 con 0x7f5c980b86b0 2024-04-25T19:13:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.413+0000 7f5c94ff9640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c780744e0 0x7f5c780769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.413+0000 7f5c97fff640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c780744e0 0x7f5c780769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.413+0000 7f5c94ff9640 1 -- 172.21.15.28:0/3168023572 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f5c90095320 con 0x7f5c980b86b0 2024-04-25T19:13:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.414+0000 7f5c97fff640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c780744e0 0x7f5c780769f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5c800090e0 tx=0x7f5c80009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.418+0000 7f5c94ff9640 1 -- 172.21.15.28:0/3168023572 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c9806f450 con 0x7f5c980b86b0 2024-04-25T19:13:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.602+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f5c98050740 con 0x7f5c780744e0 2024-04-25T19:13:42.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.603+0000 7f5c94ff9640 1 -- 172.21.15.28:0/3168023572 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5c98050740 con 0x7f5c780744e0 2024-04-25T19:13:42.605 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c780744e0 msgr2=0x7f5c780769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c780744e0 0x7f5c780769f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5c800090e0 tx=0x7f5c80009190 comp rx=0 tx=0).stop 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c980b86b0 msgr2=0x7f5c980b1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c980b86b0 0x7f5c980b1bf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f5c980d19d0 tx=0x7f5c90007bd0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 shutdown_connections 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f5c780744e0 0x7f5c780769f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c980b86b0 0x7f5c980b1bf0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f5c980c8d90 0x7f5c980c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 --2- 172.21.15.28:0/3168023572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c84004800 0x7f5c9814ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 >> 172.21.15.28:0/3168023572 conn(0x7f5c980d0450 msgr2=0x7f5c9809c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 shutdown_connections 2024-04-25T19:13:42.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.605+0000 7f5c9e385640 1 -- 172.21.15.28:0/3168023572 wait complete. 2024-04-25T19:13:42.618 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T19:13:42.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T19:13:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.731+0000 7f09ec405640 1 Processor -- start 2024-04-25T19:13:42.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.731+0000 7f09ec405640 1 -- start start 2024-04-25T19:13:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09ec405640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f09e40b47d0 0x7f09e41601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09ec405640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09e40d5690 0x7f09e406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09ec405640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b4d00 0x7f09e406e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09ec405640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f09e40d12e0 con 0x7f09e40d5690 2024-04-25T19:13:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09ec405640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f09e40d15c0 con 0x7f09e40b47d0 2024-04-25T19:13:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09ec405640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f09e40d4bb0 con 0x7f09e40b4d00 2024-04-25T19:13:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09e99a0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09e40d5690 0x7f09e406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09e99a0640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09e40d5690 0x7f09e406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58636/0 (socket says 172.21.15.28:58636) 2024-04-25T19:13:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09e99a0640 1 -- 172.21.15.28:0/190939652 learned_addr learned my addr 172.21.15.28:0/190939652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09ea1a1640 1 --2- 172.21.15.28:0/190939652 >> v2:172.21.15.53:3300/0 conn(0x7f09e40b47d0 0x7f09e41601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09ea9a2640 1 --2- 172.21.15.28:0/190939652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b4d00 0x7f09e406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09e99a0640 1 -- 172.21.15.28:0/190939652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b4d00 msgr2=0x7f09e406e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09e99a0640 1 --2- 172.21.15.28:0/190939652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b4d00 0x7f09e406e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09e99a0640 1 -- 172.21.15.28:0/190939652 >> v2:172.21.15.53:3300/0 conn(0x7f09e40b47d0 msgr2=0x7f09e41601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09e99a0640 1 --2- 172.21.15.28:0/190939652 >> v2:172.21.15.53:3300/0 conn(0x7f09e40b47d0 0x7f09e41601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.733+0000 7f09e99a0640 1 -- 172.21.15.28:0/190939652 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e40d4e90 con 0x7f09e40d5690 2024-04-25T19:13:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.734+0000 7f09e99a0640 1 --2- 172.21.15.28:0/190939652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40d5690 0x7f09e406ded0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f09e40d36b0 tx=0x7f09d402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80434805014faf8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.734+0000 7f09e919f640 1 -- 172.21.15.28:0/190939652 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4037070 con 0x7f09e40d5690 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.734+0000 7f09e919f640 1 -- 172.21.15.28:0/190939652 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d4032440 con 0x7f09e40d5690 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.734+0000 7f09e919f640 1 -- 172.21.15.28:0/190939652 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4032810 con 0x7f09e40d5690 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 -- 172.21.15.28:0/190939652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40d5690 msgr2=0x7f09e406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 --2- 172.21.15.28:0/190939652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40d5690 0x7f09e406ded0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f09e40d36b0 tx=0x7f09d402a1e0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 -- 172.21.15.28:0/190939652 shutdown_connections 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 --2- 172.21.15.28:0/190939652 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b4d00 0x7f09e406e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 --2- 172.21.15.28:0/190939652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40d5690 0x7f09e406ded0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 --2- 172.21.15.28:0/190939652 >> v2:172.21.15.53:3300/0 conn(0x7f09e40b47d0 0x7f09e41601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 -- 172.21.15.28:0/190939652 >> 172.21.15.28:0/190939652 conn(0x7f09e40d2c60 msgr2=0x7f09e40d5bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 -- 172.21.15.28:0/190939652 shutdown_connections 2024-04-25T19:13:42.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.735+0000 7f09ec405640 1 -- 172.21.15.28:0/190939652 wait complete. 2024-04-25T19:13:42.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.737+0000 7f09ec405640 1 Processor -- start 2024-04-25T19:13:42.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.737+0000 7f09ec405640 1 -- start start 2024-04-25T19:13:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ec405640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b47d0 0x7f09e41f66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ec405640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40b4d00 0x7f09e41f8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ec405640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e40d5690 0x7f09e41f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ec405640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09e40d3780 con 0x7f09e40b4d00 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ec405640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f09e4160730 con 0x7f09e40d5690 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ec405640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f09e40d3900 con 0x7f09e40b47d0 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09e99a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40b4d00 0x7f09e41f8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09e99a0640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40b4d00 0x7f09e41f8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58650/0 (socket says 172.21.15.28:58650) 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09e99a0640 1 -- 172.21.15.28:0/4273155127 learned_addr learned my addr 172.21.15.28:0/4273155127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ea1a1640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b47d0 0x7f09e41f66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ea9a2640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e40d5690 0x7f09e41f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09e99a0640 1 -- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b47d0 msgr2=0x7f09e41f66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09e99a0640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f09e40b47d0 0x7f09e41f66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09e99a0640 1 -- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e40d5690 msgr2=0x7f09e41f94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09e99a0640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e40d5690 0x7f09e41f94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09e99a0640 1 -- 172.21.15.28:0/4273155127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09e41f7ec0 con 0x7f09e40b4d00 2024-04-25T19:13:42.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.738+0000 7f09ea9a2640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09e40d5690 0x7f09e41f94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:13:42.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.739+0000 7f09e99a0640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40b4d00 0x7f09e41f8f60 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f09d402a220 tx=0x7f09d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.739+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4041020 con 0x7f09e40b4d00 2024-04-25T19:13:42.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.739+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09d403d620 con 0x7f09e40b4d00 2024-04-25T19:13:42.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.739+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09e41f81a0 con 0x7f09e40b4d00 2024-04-25T19:13:42.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.739+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09d4037040 con 0x7f09e40b4d00 2024-04-25T19:13:42.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.739+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09e41f8760 con 0x7f09e40b4d00 2024-04-25T19:13:42.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.739+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09e41636c0 con 0x7f09e40b4d00 2024-04-25T19:13:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.740+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f09d4044020 con 0x7f09e40b4d00 2024-04-25T19:13:42.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.741+0000 7f09daffd640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09c80744e0 0x7f09c80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.741+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f09d40b5c10 con 0x7f09e40b4d00 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.741+0000 7f09ea1a1640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09c80744e0 0x7f09c80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.741+0000 7f09daffd640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f09c807aa70 0x7f09c807cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.741+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09d4032440 con 0x7f09c807aa70 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.741+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=52) ==== 24+0+0 (secure 0 0 0) 0x7f09d40b60a0 con 0x7f09e40b4d00 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.741+0000 7f09ea9a2640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f09c807aa70 0x7f09c807cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.741+0000 7f09ea1a1640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09c80744e0 0x7f09c80769f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f09e40d0650 tx=0x7f09e0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.742+0000 7f09ea9a2640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f09c807aa70 0x7f09c807cf00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.743+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 <== osd.7 v2:172.21.15.53:6808/79451145 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f09d4032440 con 0x7f09c807aa70 2024-04-25T19:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.756+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 --> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f09e406a100 con 0x7f09c807aa70 2024-04-25T19:13:42.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.756+0000 7f09daffd640 1 -- 172.21.15.28:0/4273155127 <== osd.7 v2:172.21.15.53:6808/79451145 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f09e406a100 con 0x7f09c807aa70 2024-04-25T19:13:42.758 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-25T19:13:42.758 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T19:13:42.758 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-25T19:13:42.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f09c807aa70 msgr2=0x7f09c807cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.53:6808/79451145,v1:172.21.15.53:6810/79451145] conn(0x7f09c807aa70 0x7f09c807cf00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09c80744e0 msgr2=0x7f09c80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f09c80744e0 0x7f09c80769f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f09e40d0650 tx=0x7f09e0009190 comp rx=0 tx=0).stop 2024-04-25T19:13:42.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40b4d00 msgr2=0x7f09e41f8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 --2- 172.21.15.28:0/4273155127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09e40b4d00 0x7f09e41f8f60 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f09d402a220 tx=0x7f09d40046c0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ea9a2640 1 -- 172.21.15.28:0/4273155127 reap_dead start 2024-04-25T19:13:42.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 shutdown_connections 2024-04-25T19:13:42.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 >> 172.21.15.28:0/4273155127 conn(0x7f09e40d2c60 msgr2=0x7f09e40bb010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:42.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 shutdown_connections 2024-04-25T19:13:42.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.757+0000 7f09ec405640 1 -- 172.21.15.28:0/4273155127 wait complete. 2024-04-25T19:13:42.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T19:13:42.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.880+0000 7fad185c5640 1 Processor -- start 2024-04-25T19:13:42.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.880+0000 7fad185c5640 1 -- start start 2024-04-25T19:13:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.881+0000 7fad185c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad1016bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.881+0000 7fad185c5640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad100b92a0 con 0x7fad100d2d10 2024-04-25T19:13:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad185c5640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad1009cea0 con 0x7fad100b3d00 2024-04-25T19:13:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad185c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fad100d28d0 con 0x7fad101696c0 2024-04-25T19:13:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad16361640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fad100d2d10 0x7fad100d21c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38952/0 (socket says 172.21.15.28:38952) 2024-04-25T19:13:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad16361640 1 -- 172.21.15.28:0/2758616541 learned_addr learned my addr 172.21.15.28:0/2758616541 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad16b62640 1 --2- 172.21.15.28:0/2758616541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad1016bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad1535f640 1 -- 172.21.15.28:0/2758616541 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1702573215 0 0) 0x7fad100b92a0 con 0x7fad100d2d10 2024-04-25T19:13:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad1535f640 1 -- 172.21.15.28:0/2758616541 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facfc003520 con 0x7fad100d2d10 2024-04-25T19:13:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad1535f640 1 -- 172.21.15.28:0/2758616541 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2981551926 0 0) 0x7fad1009cea0 con 0x7fad100b3d00 2024-04-25T19:13:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad1535f640 1 -- 172.21.15.28:0/2758616541 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad100b92a0 con 0x7fad100b3d00 2024-04-25T19:13:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad16b62640 1 -- 172.21.15.28:0/2758616541 >> v1:172.21.15.53:6789/0 conn(0x7fad100b3d00 legacy=0x7fad100b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad16b62640 1 -- 172.21.15.28:0/2758616541 >> v1:172.21.15.28:6789/0 conn(0x7fad100d2d10 legacy=0x7fad100d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.882+0000 7fad16b62640 1 -- 172.21.15.28:0/2758616541 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad1006e8e0 con 0x7fad101696c0 2024-04-25T19:13:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.883+0000 7fad16b62640 1 --2- 172.21.15.28:0/2758616541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad1016bae0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fad08009dc0 tx=0x7fad0802a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9de25b70783d7ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.883+0000 7fad1535f640 1 -- 172.21.15.28:0/2758616541 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad0802ae60 con 0x7fad101696c0 2024-04-25T19:13:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.883+0000 7fad1535f640 1 -- 172.21.15.28:0/2758616541 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad08007740 con 0x7fad101696c0 2024-04-25T19:13:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.883+0000 7fad1535f640 1 -- 172.21.15.28:0/2758616541 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad08007b10 con 0x7fad101696c0 2024-04-25T19:13:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.884+0000 7fad185c5640 1 -- 172.21.15.28:0/2758616541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 msgr2=0x7fad1016bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.884+0000 7fad185c5640 1 --2- 172.21.15.28:0/2758616541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad1016bae0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fad08009dc0 tx=0x7fad0802a640 comp rx=0 tx=0).stop 2024-04-25T19:13:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.884+0000 7fad185c5640 1 -- 172.21.15.28:0/2758616541 shutdown_connections 2024-04-25T19:13:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.884+0000 7fad185c5640 1 --2- 172.21.15.28:0/2758616541 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad1016bae0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.884+0000 7fad185c5640 1 -- 172.21.15.28:0/2758616541 >> 172.21.15.28:0/2758616541 conn(0x7fad100d0450 msgr2=0x7fad100d0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.884+0000 7fad185c5640 1 -- 172.21.15.28:0/2758616541 shutdown_connections 2024-04-25T19:13:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.884+0000 7fad185c5640 1 -- 172.21.15.28:0/2758616541 wait complete. 2024-04-25T19:13:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.886+0000 7fad185c5640 1 Processor -- start 2024-04-25T19:13:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.886+0000 7fad185c5640 1 -- start start 2024-04-25T19:13:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad185c5640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad100b3d00 0x7fad101600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad185c5640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad100d2d10 0x7fad100b0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad185c5640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad101f9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad185c5640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad1006ebc0 con 0x7fad100d2d10 2024-04-25T19:13:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad185c5640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fad1006e690 con 0x7fad100b3d00 2024-04-25T19:13:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad185c5640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fad100d3120 con 0x7fad101696c0 2024-04-25T19:13:42.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad15b60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad100d2d10 0x7fad100b0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad15b60640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad100d2d10 0x7fad100b0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58652/0 (socket says 172.21.15.28:58652) 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad15b60640 1 -- 172.21.15.28:0/1367035448 learned_addr learned my addr 172.21.15.28:0/1367035448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad16361640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad100b3d00 0x7fad101600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.887+0000 7fad16b62640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad101f9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad15b60640 1 -- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 msgr2=0x7fad101f9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad15b60640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad101f9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad15b60640 1 -- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad100b3d00 msgr2=0x7fad101600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad15b60640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad100b3d00 0x7fad101600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad15b60640 1 -- 172.21.15.28:0/1367035448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad100b0f70 con 0x7fad100d2d10 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad16b62640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad101f9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad15b60640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad100d2d10 0x7fad100b0a30 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fad0400c3f0 tx=0x7fad0400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad02ffd640 1 -- 172.21.15.28:0/1367035448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04017070 con 0x7fad100d2d10 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad02ffd640 1 -- 172.21.15.28:0/1367035448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad040052f0 con 0x7fad100d2d10 2024-04-25T19:13:42.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad10161440 con 0x7fad100d2d10 2024-04-25T19:13:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad02ffd640 1 -- 172.21.15.28:0/1367035448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04003830 con 0x7fad100d2d10 2024-04-25T19:13:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.888+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad101fa2b0 con 0x7fad100d2d10 2024-04-25T19:13:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.889+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad10050740 con 0x7fad100d2d10 2024-04-25T19:13:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.890+0000 7fad02ffd640 1 -- 172.21.15.28:0/1367035448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fad0401a070 con 0x7fad100d2d10 2024-04-25T19:13:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.890+0000 7fad02ffd640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7facf40744b0 0x7facf40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.891+0000 7fad16361640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7facf40744b0 0x7facf40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.891+0000 7fad02ffd640 1 -- 172.21.15.28:0/1367035448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7fad040943f0 con 0x7fad100d2d10 2024-04-25T19:13:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.891+0000 7fad16361640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7facf40744b0 0x7facf40769c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0011e0 tx=0x7fad0c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:42.895+0000 7fad02ffd640 1 -- 172.21.15.28:0/1367035448 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad0405ed30 con 0x7fad100d2d10 2024-04-25T19:13:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.081+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7fad100d0ab0 con 0x7facf40744b0 2024-04-25T19:13:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.081+0000 7fad02ffd640 1 -- 172.21.15.28:0/1367035448 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fad100d0ab0 con 0x7facf40744b0 2024-04-25T19:13:43.083 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T19:13:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7facf40744b0 msgr2=0x7facf40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7facf40744b0 0x7facf40769c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0011e0 tx=0x7fad0c009040 comp rx=0 tx=0).stop 2024-04-25T19:13:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad100d2d10 msgr2=0x7fad100b0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad100d2d10 0x7fad100b0a30 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fad0400c3f0 tx=0x7fad0400c8b0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 shutdown_connections 2024-04-25T19:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7facf40744b0 0x7facf40769c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fad101696c0 0x7fad101f9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad100d2d10 0x7fad100b0a30 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 --2- 172.21.15.28:0/1367035448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad100b3d00 0x7fad101600c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 >> 172.21.15.28:0/1367035448 conn(0x7fad100d0450 msgr2=0x7fad100b4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 shutdown_connections 2024-04-25T19:13:43.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.083+0000 7fad185c5640 1 -- 172.21.15.28:0/1367035448 wait complete. 2024-04-25T19:13:43.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:13:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.206+0000 7f711d40c640 1 Processor -- start 2024-04-25T19:13:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.206+0000 7f711d40c640 1 -- start start 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.207+0000 7f711d40c640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f711806ebd0 0x7f71180af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.207+0000 7f711d40c640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f71181604b0 0x7f711809b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.207+0000 7f711d40c640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f711809be90 con 0x7f711809ba80 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.207+0000 7f711d40c640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7f71180b00b0 con 0x7f711806ebd0 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.207+0000 7f711d40c640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f71180b0670 con 0x7f71181604b0 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.207+0000 7f7116ffd640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f711806ebd0 0x7f71180af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.207+0000 7f71177fe640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f711809ba80 0x7f7118163f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38964/0 (socket says 172.21.15.28:38964) 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.208+0000 7f71177fe640 1 -- 172.21.15.28:0/4203532759 learned_addr learned my addr 172.21.15.28:0/4203532759 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.208+0000 7f71167fc640 1 --2- 172.21.15.28:0/4203532759 >> v2:172.21.15.53:3300/0 conn(0x7f71181604b0 0x7f711809b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.208+0000 7f7116ffd640 1 -- 172.21.15.28:0/4203532759 >> v2:172.21.15.53:3300/0 conn(0x7f71181604b0 msgr2=0x7f711809b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:13:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.208+0000 7f7116ffd640 1 --2- 172.21.15.28:0/4203532759 >> v2:172.21.15.53:3300/0 conn(0x7f71181604b0 0x7f711809b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.208+0000 7f7116ffd640 1 -- 172.21.15.28:0/4203532759 >> v1:172.21.15.28:6789/0 conn(0x7f711809ba80 legacy=0x7f7118163f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.208+0000 7f7116ffd640 1 -- 172.21.15.28:0/4203532759 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71180d2f20 con 0x7f711806ebd0 2024-04-25T19:13:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f7116ffd640 1 --2- 172.21.15.28:0/4203532759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f711806ebd0 0x7f71180af6a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f711809c260 tx=0x7f710802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ddd500e2208253 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f7115ffb640 1 -- 172.21.15.28:0/4203532759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108037070 con 0x7f711806ebd0 2024-04-25T19:13:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f7115ffb640 1 -- 172.21.15.28:0/4203532759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7108032440 con 0x7f711806ebd0 2024-04-25T19:13:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f7115ffb640 1 -- 172.21.15.28:0/4203532759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7108032810 con 0x7f711806ebd0 2024-04-25T19:13:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f711d40c640 1 -- 172.21.15.28:0/4203532759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f711806ebd0 msgr2=0x7f71180af6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f711d40c640 1 --2- 172.21.15.28:0/4203532759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f711806ebd0 0x7f71180af6a0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f711809c260 tx=0x7f710802a1a0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f711d40c640 1 -- 172.21.15.28:0/4203532759 shutdown_connections 2024-04-25T19:13:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f711d40c640 1 --2- 172.21.15.28:0/4203532759 >> v2:172.21.15.53:3300/0 conn(0x7f71181604b0 0x7f711809b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f711d40c640 1 --2- 172.21.15.28:0/4203532759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f711806ebd0 0x7f71180af6a0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f711d40c640 1 -- 172.21.15.28:0/4203532759 >> 172.21.15.28:0/4203532759 conn(0x7f71180b7560 msgr2=0x7f71180b7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f711d40c640 1 -- 172.21.15.28:0/4203532759 shutdown_connections 2024-04-25T19:13:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.209+0000 7f711d40c640 1 -- 172.21.15.28:0/4203532759 wait complete. 2024-04-25T19:13:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.211+0000 7f711d40c640 1 Processor -- start 2024-04-25T19:13:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.211+0000 7f711d40c640 1 -- start start 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f711d40c640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f711806ebd0 0x7f7118158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f711d40c640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f711809ba80 0x7f7118151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f711d40c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71181604b0 0x7f71180b4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f711d40c640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f71180b0300 con 0x7f71181604b0 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f711d40c640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f71180d34e0 con 0x7f711809ba80 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f711d40c640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f71180b3ce0 con 0x7f711806ebd0 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71181604b0 0x7f71180b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71181604b0 0x7f71180b4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58674/0 (socket says 172.21.15.28:58674) 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 -- 172.21.15.28:0/292950255 learned_addr learned my addr 172.21.15.28:0/292950255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:13:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f7116ffd640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f711806ebd0 0x7f7118158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71167fc640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f711809ba80 0x7f7118151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 -- 172.21.15.28:0/292950255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f711806ebd0 msgr2=0x7f7118158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f711806ebd0 0x7f7118158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 -- 172.21.15.28:0/292950255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f711809ba80 msgr2=0x7f7118151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f711809ba80 0x7f7118151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 -- 172.21.15.28:0/292950255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7118151a90 con 0x7f71181604b0 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f71177fe640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71181604b0 0x7f71180b4350 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f711000ca80 tx=0x7f711000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.213+0000 7f70ff7fe640 1 -- 172.21.15.28:0/292950255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110017070 con 0x7f71181604b0 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.214+0000 7f70ff7fe640 1 -- 172.21.15.28:0/292950255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f711000ac30 con 0x7f71181604b0 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.214+0000 7f70ff7fe640 1 -- 172.21.15.28:0/292950255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7110012400 con 0x7f71181604b0 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.214+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711815f2a0 con 0x7f71181604b0 2024-04-25T19:13:43.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.214+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f711815e4e0 con 0x7f71181604b0 2024-04-25T19:13:43.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.215+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f711815df70 con 0x7f71181604b0 2024-04-25T19:13:43.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.215+0000 7f70ff7fe640 1 -- 172.21.15.28:0/292950255 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f711001a070 con 0x7f71181604b0 2024-04-25T19:13:43.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.216+0000 7f70ff7fe640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71040744b0 0x7f71040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:13:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.216+0000 7f7116ffd640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71040744b0 0x7f71040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:13:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.216+0000 7f70ff7fe640 1 -- 172.21.15.28:0/292950255 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f7110094d60 con 0x7f71181604b0 2024-04-25T19:13:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.216+0000 7f7116ffd640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71040744b0 0x7f71040769c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7108034000 tx=0x7f710802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:13:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.220+0000 7f70ff7fe640 1 -- 172.21.15.28:0/292950255 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7118151a90 con 0x7f71181604b0 2024-04-25T19:13:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.402+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f71180b8150 con 0x7f71040744b0 2024-04-25T19:13:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.403+0000 7f70ff7fe640 1 -- 172.21.15.28:0/292950255 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13357 (secure 0 0 0) 0x7f71180b8150 con 0x7f71040744b0 2024-04-25T19:13:43.405 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:13:43.405 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71040744b0 msgr2=0x7f71040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71040744b0 0x7f71040769c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7108034000 tx=0x7f710802e000 comp rx=0 tx=0).stop 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71181604b0 msgr2=0x7f71180b4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71181604b0 0x7f71180b4350 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f711000ca80 tx=0x7f711000cf40 comp rx=0 tx=0).stop 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 shutdown_connections 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f71040744b0 0x7f71040769c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71181604b0 0x7f71180b4350 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f711809ba80 0x7f7118151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 --2- 172.21.15.28:0/292950255 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f711806ebd0 0x7f7118158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.404+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 >> 172.21.15.28:0/292950255 conn(0x7f71180b7560 msgr2=0x7f711815f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:13:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.405+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 shutdown_connections 2024-04-25T19:13:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:13:43.405+0000 7f711d40c640 1 -- 172.21.15.28:0/292950255 wait complete. 2024-04-25T19:13:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-04-25T19:13:42.865998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":255492,"kb_used_data":6024,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719228,"statfs":{"total":773094113280,"available":772832489472,"internally_reserved":0,"allocated":6168576,"data_stored":3452920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6291456,"num_objects":-2,"num_object_clones":0,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":-16,"num_read_kb":-9,"num_write":-8,"num_write_kb":-6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.817906"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":169,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-25T19:13:34.056255+0000","last_change":"2024-04-25T19:11:17.908778+0000","last_active":"2024-04-25T19:13:34.056255+0000","last_peered":"2024-04-25T19:13:34.056255+0000","last_clean":"2024-04-25T19:13:34.056255+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:13:34.056255+0000","last_undegraded":"2024-04-25T19:13:34.056255+0000","last_fullsized":"2024-04-25T19:13:34.056255+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T19:11:16.869294+0000","last_clean_scrub_stamp":"2024-04-25T19:11:16.869294+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:49:43.059018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32372,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339468,"statfs":{"total":96636764160,"available":96603615232,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32368,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339472,"statfs":{"total":96636764160,"available":96603619328,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:13:43.421 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T19:11:16.869294+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=19, tm_min=11, tm_sec=16, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=19, tm_min=13, tm_sec=38, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T19:13:43.421 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T19:14:02.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:14:02.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:14:03.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T19:14:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.546+0000 7f7e466a7640 1 Processor -- start 2024-04-25T19:14:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.546+0000 7f7e466a7640 1 -- start start 2024-04-25T19:14:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e466a7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e40164990 0x7f7e4015dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e466a7640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7e401606f0 0x7f7e40160b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e466a7640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e401614a0 con 0x7f7e400b7660 2024-04-25T19:14:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e466a7640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f7e40163eb0 con 0x7f7e401606f0 2024-04-25T19:14:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e466a7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7e40164190 con 0x7f7e40164990 2024-04-25T19:14:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e44c44640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7e401606f0 0x7f7e40160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e3ffff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e40164990 0x7f7e4015dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e3ffff640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e40164990 0x7f7e4015dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3301/0 says I am v2:172.21.15.28:45108/0 (socket says 172.21.15.28:45108) 2024-04-25T19:14:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e44c44640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7e401606f0 0x7f7e40160b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.28:59130/0 (socket says 172.21.15.28:59130) 2024-04-25T19:14:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e3ffff640 1 -- 172.21.15.28:0/3671936270 learned_addr learned my addr 172.21.15.28:0/3671936270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:14:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.548+0000 7f7e3f7fe640 1 --1- 172.21.15.28:0/3671936270 >> v1:172.21.15.53:6789/0 conn(0x7f7e400b7660 0x7f7e400b7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.28:38188/0 (socket says 172.21.15.28:38188) 2024-04-25T19:14:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.549+0000 7f7e3ffff640 1 -- 172.21.15.28:0/3671936270 >> v2:172.21.15.53:3300/0 conn(0x7f7e401606f0 msgr2=0x7f7e40160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:14:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.549+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/3671936270 >> v2:172.21.15.53:3300/0 conn(0x7f7e401606f0 0x7f7e40160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.549+0000 7f7e3effd640 1 -- 172.21.15.28:0/3671936270 <== mon.1 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1707326649 0 0) 0x7f7e401614a0 con 0x7f7e400b7660 2024-04-25T19:14:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.549+0000 7f7e3ffff640 1 -- 172.21.15.28:0/3671936270 >> v1:172.21.15.53:6789/0 conn(0x7f7e400b7660 legacy=0x7f7e400b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.549+0000 7f7e3ffff640 1 -- 172.21.15.28:0/3671936270 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e40164470 con 0x7f7e40164990 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.549+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/3671936270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e40164990 0x7f7e4015dcb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009870 tx=0x7f7e2802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0f24c212314a14 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e3effd640 1 -- 172.21.15.28:0/3671936270 <== mon.2 v2:172.21.15.53:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28037070 con 0x7f7e40164990 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e3effd640 1 -- 172.21.15.28:0/3671936270 <== mon.2 v2:172.21.15.53:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e28032440 con 0x7f7e40164990 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e3effd640 1 -- 172.21.15.28:0/3671936270 <== mon.2 v2:172.21.15.53:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e28032810 con 0x7f7e40164990 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e466a7640 1 -- 172.21.15.28:0/3671936270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e40164990 msgr2=0x7f7e4015dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e466a7640 1 --2- 172.21.15.28:0/3671936270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e40164990 0x7f7e4015dcb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7e28009870 tx=0x7f7e2802a270 comp rx=0 tx=0).stop 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e466a7640 1 -- 172.21.15.28:0/3671936270 shutdown_connections 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e466a7640 1 --2- 172.21.15.28:0/3671936270 >> v2:172.21.15.53:3300/0 conn(0x7f7e401606f0 0x7f7e40160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e466a7640 1 --2- 172.21.15.28:0/3671936270 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e40164990 0x7f7e4015dcb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e466a7640 1 -- 172.21.15.28:0/3671936270 >> 172.21.15.28:0/3671936270 conn(0x7f7e400d2840 msgr2=0x7f7e400d2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e466a7640 1 -- 172.21.15.28:0/3671936270 shutdown_connections 2024-04-25T19:14:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.550+0000 7f7e466a7640 1 -- 172.21.15.28:0/3671936270 wait complete. 2024-04-25T19:14:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.552+0000 7f7e466a7640 1 Processor -- start 2024-04-25T19:14:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.553+0000 7f7e466a7640 1 -- start start 2024-04-25T19:14:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e466a7640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e400b7660 0x7f7e400cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e466a7640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e401606f0 0x7f7e400cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e466a7640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40164990 0x7f7e400cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e466a7640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e400b4750 con 0x7f7e40164990 2024-04-25T19:14:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e466a7640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f7e400b45d0 con 0x7f7e400b7660 2024-04-25T19:14:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e466a7640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f7e400b48d0 con 0x7f7e401606f0 2024-04-25T19:14:03.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e44c44640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40164990 0x7f7e400cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e44c44640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40164990 0x7f7e400cc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45596/0 (socket says 172.21.15.28:45596) 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e44c44640 1 -- 172.21.15.28:0/2422873205 learned_addr learned my addr 172.21.15.28:0/2422873205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e400b7660 0x7f7e400cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e3f7fe640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e401606f0 0x7f7e400cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e44c44640 1 -- 172.21.15.28:0/2422873205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e401606f0 msgr2=0x7f7e400cf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e44c44640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e401606f0 0x7f7e400cf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e44c44640 1 -- 172.21.15.28:0/2422873205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e400b7660 msgr2=0x7f7e400cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e44c44640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e400b7660 0x7f7e400cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.554+0000 7f7e44c44640 1 -- 172.21.15.28:0/2422873205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e400ccd30 con 0x7f7e40164990 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.555+0000 7f7e44c44640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40164990 0x7f7e400cc350 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009cb0 tx=0x7f7e3800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.555+0000 7f7e3cff9640 1 -- 172.21.15.28:0/2422873205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38018020 con 0x7f7e40164990 2024-04-25T19:14:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.555+0000 7f7e3cff9640 1 -- 172.21.15.28:0/2422873205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e38014480 con 0x7f7e40164990 2024-04-25T19:14:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.555+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e400cfab0 con 0x7f7e40164990 2024-04-25T19:14:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.555+0000 7f7e3cff9640 1 -- 172.21.15.28:0/2422873205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e38014870 con 0x7f7e40164990 2024-04-25T19:14:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.555+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e400ccfa0 con 0x7f7e40164990 2024-04-25T19:14:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.556+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e400b63a0 con 0x7f7e40164990 2024-04-25T19:14:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.557+0000 7f7e3cff9640 1 -- 172.21.15.28:0/2422873205 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7e38005d10 con 0x7f7e40164990 2024-04-25T19:14:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.557+0000 7f7e3cff9640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e200744b0 0x7f7e200769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.557+0000 7f7e3cff9640 1 -- 172.21.15.28:0/2422873205 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f7e38095cd0 con 0x7f7e40164990 2024-04-25T19:14:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.557+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e200744b0 0x7f7e200769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.558+0000 7f7e3ffff640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e200744b0 0x7f7e200769c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7e28034000 tx=0x7f7e2802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.562+0000 7f7e3cff9640 1 -- 172.21.15.28:0/2422873205 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7e400b63a0 con 0x7f7e40164990 2024-04-25T19:14:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.745+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 --> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7e400c9810 con 0x7f7e200744b0 2024-04-25T19:14:03.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.746+0000 7f7e3cff9640 1 -- 172.21.15.28:0/2422873205 <== mgr.4110 v2:172.21.15.28:6832/2354405395 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13344 (secure 0 0 0) 0x7f7e400c9810 con 0x7f7e200744b0 2024-04-25T19:14:03.748 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:14:03.749 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-25T19:14:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.747+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e200744b0 msgr2=0x7f7e200769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e200744b0 0x7f7e200769c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7e28034000 tx=0x7f7e2802e000 comp rx=0 tx=0).stop 2024-04-25T19:14:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40164990 msgr2=0x7f7e400cc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40164990 0x7f7e400cc350 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009cb0 tx=0x7f7e3800c490 comp rx=0 tx=0).stop 2024-04-25T19:14:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 shutdown_connections 2024-04-25T19:14:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f7e200744b0 0x7f7e200769c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e40164990 0x7f7e400cc350 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f7e401606f0 0x7f7e400cf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 --2- 172.21.15.28:0/2422873205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e400b7660 0x7f7e400cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 >> 172.21.15.28:0/2422873205 conn(0x7f7e400d2840 msgr2=0x7f7e400bb380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:14:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 shutdown_connections 2024-04-25T19:14:03.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.748+0000 7f7e466a7640 1 -- 172.21.15.28:0/2422873205 wait complete. 2024-04-25T19:14:03.761 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-04-25T19:14:02.870417+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":255480,"kb_used_data":6024,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719240,"statfs":{"total":773094113280,"available":772832501760,"internally_reserved":0,"allocated":6168576,"data_stored":3452920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":255315507},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002605"},"pg_stats":[{"pgid":"1.0","version":"52'74","reported_seq":180,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-25T19:13:43.254614+0000","last_change":"2024-04-25T19:13:43.254659+0000","last_active":"2024-04-25T19:13:43.254614+0000","last_peered":"2024-04-25T19:13:43.254614+0000","last_clean":"2024-04-25T19:13:43.254614+0000","last_became_active":"2024-04-25T19:11:17.907816+0000","last_became_peered":"2024-04-25T19:11:17.907816+0000","last_unstale":"2024-04-25T19:13:43.254614+0000","last_undegraded":"2024-04-25T19:13:43.254614+0000","last_fullsized":"2024-04-25T19:13:43.254614+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"52'74","last_scrub_stamp":"2024-04-25T19:13:43.254522+0000","last_deep_scrub":"52'74","last_deep_scrub_stamp":"2024-04-25T19:13:43.254522+0000","last_clean_scrub_stamp":"2024-04-25T19:13:43.254522+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:22:37.122442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31792,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340048,"statfs":{"total":96636764160,"available":96604209152,"internally_reserved":0,"allocated":622592,"data_stored":284023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32364,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339476,"statfs":{"total":96636764160,"available":96603623424,"internally_reserved":0,"allocated":1216512,"data_stored":874391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T19:14:03.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T19:14:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.871+0000 7fd292dce640 1 Processor -- start 2024-04-25T19:14:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.871+0000 7fd292dce640 1 -- start start 2024-04-25T19:14:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.872+0000 7fd292dce640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd28c160250 0x7fd28c0b4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.872+0000 7fd292dce640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd28c0d2ca0 0x7fd28c0d44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.872+0000 7fd292dce640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd28c164b20 con 0x7fd28c0b81a0 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.872+0000 7fd292dce640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 -- 0x7fd28c0b4f50 con 0x7fd28c160250 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.872+0000 7fd292dce640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7fd28c163d60 con 0x7fd28c0d2ca0 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.872+0000 7fd290b6a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd28c160250 0x7fd28c0b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd28bfff640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd28c0b81a0 0x7fd28c0d2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36258/0 (socket says 172.21.15.28:36258) 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd290b6a640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd28c160250 0x7fd28c0b4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45608/0 (socket says 172.21.15.28:45608) 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd28bfff640 1 -- 172.21.15.28:0/3064136495 learned_addr learned my addr 172.21.15.28:0/3064136495 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:14:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd29136b640 1 --2- 172.21.15.28:0/3064136495 >> v2:172.21.15.53:3300/0 conn(0x7fd28c0d2ca0 0x7fd28c0d44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd290b6a640 1 -- 172.21.15.28:0/3064136495 >> v2:172.21.15.53:3300/0 conn(0x7fd28c0d2ca0 msgr2=0x7fd28c0d44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd290b6a640 1 --2- 172.21.15.28:0/3064136495 >> v2:172.21.15.53:3300/0 conn(0x7fd28c0d2ca0 0x7fd28c0d44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd290b6a640 1 -- 172.21.15.28:0/3064136495 >> v1:172.21.15.28:6789/0 conn(0x7fd28c0b81a0 legacy=0x7fd28c0d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd290b6a640 1 -- 172.21.15.28:0/3064136495 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c164040 con 0x7fd28c160250 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd28b7fe640 1 -- 172.21.15.28:0/3064136495 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 697820375 0 0) 0x7fd28c164b20 con 0x7fd28c0b81a0 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.873+0000 7fd29136b640 1 --2- 172.21.15.28:0/3064136495 >> v2:172.21.15.53:3300/0 conn(0x7fd28c0d2ca0 0x7fd28c0d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.874+0000 7fd290b6a640 1 --2- 172.21.15.28:0/3064136495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c160250 0x7fd28c0b4690 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0d3890 tx=0x7fd28002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7ddff952033db3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.874+0000 7fd28b7fe640 1 -- 172.21.15.28:0/3064136495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280037070 con 0x7fd28c160250 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.874+0000 7fd28b7fe640 1 -- 172.21.15.28:0/3064136495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd280032440 con 0x7fd28c160250 2024-04-25T19:14:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.874+0000 7fd28b7fe640 1 -- 172.21.15.28:0/3064136495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28002ae60 con 0x7fd28c160250 2024-04-25T19:14:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.874+0000 7fd292dce640 1 -- 172.21.15.28:0/3064136495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c160250 msgr2=0x7fd28c0b4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.874+0000 7fd292dce640 1 --2- 172.21.15.28:0/3064136495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c160250 0x7fd28c0b4690 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd28c0d3890 tx=0x7fd28002a110 comp rx=0 tx=0).stop 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.875+0000 7fd292dce640 1 -- 172.21.15.28:0/3064136495 shutdown_connections 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.875+0000 7fd292dce640 1 --2- 172.21.15.28:0/3064136495 >> v2:172.21.15.53:3300/0 conn(0x7fd28c0d2ca0 0x7fd28c0d44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.875+0000 7fd292dce640 1 --2- 172.21.15.28:0/3064136495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c160250 0x7fd28c0b4690 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.875+0000 7fd292dce640 1 -- 172.21.15.28:0/3064136495 >> 172.21.15.28:0/3064136495 conn(0x7fd28c06ee30 msgr2=0x7fd28c06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.875+0000 7fd292dce640 1 -- 172.21.15.28:0/3064136495 shutdown_connections 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.875+0000 7fd292dce640 1 -- 172.21.15.28:0/3064136495 wait complete. 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.876+0000 7fd292dce640 1 Processor -- start 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.876+0000 7fd292dce640 1 -- start start 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd292dce640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c0b81a0 0x7fd28c1635f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd292dce640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd28c160250 0x7fd28c14afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd292dce640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd28c162110 0x7fd28c15e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd292dce640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c0b4bd0 con 0x7fd28c0b81a0 2024-04-25T19:14:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd292dce640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7fd28c164600 con 0x7fd28c162110 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd292dce640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7fd28c0d49f0 con 0x7fd28c160250 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c0b81a0 0x7fd28c1635f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c0b81a0 0x7fd28c1635f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45620/0 (socket says 172.21.15.28:45620) 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 -- 172.21.15.28:0/2764281450 learned_addr learned my addr 172.21.15.28:0/2764281450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd29136b640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd28c162110 0x7fd28c15e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd28bfff640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd28c160250 0x7fd28c14afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 -- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd28c160250 msgr2=0x7fd28c14afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd28c160250 0x7fd28c14afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 -- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd28c162110 msgr2=0x7fd28c15e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd28c162110 0x7fd28c15e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 -- 172.21.15.28:0/2764281450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28c0cd0c0 con 0x7fd28c0b81a0 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd29136b640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd28c162110 0x7fd28c15e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.878+0000 7fd290b6a640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c0b81a0 0x7fd28c1635f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd280035040 tx=0x7fd2800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:03.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.879+0000 7fd2897fa640 1 -- 172.21.15.28:0/2764281450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280037050 con 0x7fd28c0b81a0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.879+0000 7fd2897fa640 1 -- 172.21.15.28:0/2764281450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd280038040 con 0x7fd28c0b81a0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.879+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28c1628f0 con 0x7fd28c0b81a0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.879+0000 7fd2897fa640 1 -- 172.21.15.28:0/2764281450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd280004470 con 0x7fd28c0b81a0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.879+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28c14bb60 con 0x7fd28c0b81a0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.880+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd28c050740 con 0x7fd28c0b81a0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.880+0000 7fd2897fa640 1 -- 172.21.15.28:0/2764281450 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd280030980 con 0x7fd28c0b81a0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.881+0000 7fd2897fa640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2700744b0 0x7fd2700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.881+0000 7fd2897fa640 1 -- 172.21.15.28:0/2764281450 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7fd2800b5bb0 con 0x7fd28c0b81a0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.881+0000 7fd28bfff640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2700744b0 0x7fd2700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:03.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.881+0000 7fd28bfff640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2700744b0 0x7fd2700769c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd28c15f570 tx=0x7fd278007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:03.886+0000 7fd2897fa640 1 -- 172.21.15.28:0/2764281450 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd280080580 con 0x7fd28c0b81a0 2024-04-25T19:14:04.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.070+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fd28c0d3fd0 con 0x7fd28c0b81a0 2024-04-25T19:14:04.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.074+0000 7fd2897fa640 1 -- 172.21.15.28:0/2764281450 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7fd28007fe80 con 0x7fd28c0b81a0 2024-04-25T19:14:04.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.074+0000 7fd2897fa640 1 -- 172.21.15.28:0/2764281450 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fd2800093e0 con 0x7fd28c0b81a0 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.076+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2700744b0 msgr2=0x7fd2700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.076+0000 7fd292dce640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2700744b0 0x7fd2700769c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd28c15f570 tx=0x7fd278007c00 comp rx=0 tx=0).stop 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.076+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c0b81a0 msgr2=0x7fd28c1635f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.076+0000 7fd292dce640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c0b81a0 0x7fd28c1635f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd280035040 tx=0x7fd2800046c0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.077+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 shutdown_connections 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.077+0000 7fd292dce640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7fd2700744b0 0x7fd2700769c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.077+0000 7fd292dce640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd28c162110 0x7fd28c15e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.077+0000 7fd292dce640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7fd28c160250 0x7fd28c14afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.077+0000 7fd292dce640 1 --2- 172.21.15.28:0/2764281450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd28c0b81a0 0x7fd28c1635f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.077+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 >> 172.21.15.28:0/2764281450 conn(0x7fd28c06ee30 msgr2=0x7fd28c0d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:14:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.077+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 shutdown_connections 2024-04-25T19:14:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.077+0000 7fd292dce640 1 -- 172.21.15.28:0/2764281450 wait complete. 2024-04-25T19:14:04.090 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T19:14:04.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T19:14:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.201+0000 7f3954900640 1 Processor -- start 2024-04-25T19:14:04.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.201+0000 7f3954900640 1 -- start start 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f3954900640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395006e5e0 0x7f3950161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f3954900640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f395015f7d0 0x7f395015fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f3954900640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3950163e80 con 0x7f39500b5130 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f3954900640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 -- 0x7f3950163c30 con 0x7f395015f7d0 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f3954900640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f3950164a90 con 0x7f395006e5e0 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394e59c640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f39500b5130 0x7f395006ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36270/0 (socket says 172.21.15.28:36270) 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394e59c640 1 -- 172.21.15.28:0/501515443 learned_addr learned my addr 172.21.15.28:0/501515443 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3042130879 0 0) 0x7f3950163e80 con 0x7f39500b5130 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f393c003520 con 0x7f39500b5130 2024-04-25T19:14:04.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394ed9d640 1 --2- 172.21.15.28:0/501515443 >> v2:172.21.15.53:3300/0 conn(0x7f395015f7d0 0x7f395015fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394dd9b640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395006e5e0 0x7f3950161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1598518374 0 0) 0x7f393c003520 con 0x7f39500b5130 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395006e5e0 msgr2=0x7f3950161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394d59a640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f395006e5e0 0x7f3950161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 >> v2:172.21.15.53:3300/0 conn(0x7f395015f7d0 msgr2=0x7f395015fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394d59a640 1 --2- 172.21.15.28:0/501515443 >> v2:172.21.15.53:3300/0 conn(0x7f395015f7d0 0x7f395015fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.203+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3950164160 con 0x7f39500b5130 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 461656568 0 0) 0x7f3948001670 con 0x7f39500b5130 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 >> v1:172.21.15.28:6789/0 conn(0x7f39500b5130 legacy=0x7f395006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394d59a640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f393c0039b0 0x7f393c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394d59a640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f393c004800 0x7f393c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394d59a640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f393c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3950163c30 con 0x7f393c005150 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3950164a90 con 0x7f393c0039b0 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f393c00bf00 con 0x7f393c004800 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394ed9d640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f393c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394e59c640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f393c0039b0 0x7f393c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394dd9b640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f393c004800 0x7f393c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394ed9d640 1 -- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f393c004800 msgr2=0x7f393c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394ed9d640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f393c004800 0x7f393c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394ed9d640 1 -- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f393c0039b0 msgr2=0x7f393c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T19:14:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394ed9d640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f393c0039b0 0x7f393c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.204+0000 7f394ed9d640 1 -- 172.21.15.28:0/501515443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f393c003520 con 0x7f393c005150 2024-04-25T19:14:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.205+0000 7f394ed9d640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f393c00b9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3950163b90 tx=0x7f393c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c57b3201204ce37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.205+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940009c10 con 0x7f393c005150 2024-04-25T19:14:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.205+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3940036070 con 0x7f393c005150 2024-04-25T19:14:04.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.205+0000 7f394d59a640 1 -- 172.21.15.28:0/501515443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3940031830 con 0x7f393c005150 2024-04-25T19:14:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.206+0000 7f3954900640 1 -- 172.21.15.28:0/501515443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 msgr2=0x7f393c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.206+0000 7f3954900640 1 --2- 172.21.15.28:0/501515443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f393c00b9c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f3950163b90 tx=0x7f393c005830 comp rx=0 tx=0).stop 2024-04-25T19:14:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.206+0000 7f394ed9d640 1 -- 172.21.15.28:0/501515443 reap_dead start 2024-04-25T19:14:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.206+0000 7f3954900640 1 -- 172.21.15.28:0/501515443 shutdown_connections 2024-04-25T19:14:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.206+0000 7f3954900640 1 -- 172.21.15.28:0/501515443 >> 172.21.15.28:0/501515443 conn(0x7f39500b7ad0 msgr2=0x7f39500b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:14:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.206+0000 7f3954900640 1 -- 172.21.15.28:0/501515443 shutdown_connections 2024-04-25T19:14:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.206+0000 7f3954900640 1 -- 172.21.15.28:0/501515443 wait complete. 2024-04-25T19:14:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.208+0000 7f3954900640 1 Processor -- start 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.208+0000 7f3954900640 1 -- start start 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f3954900640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f3950151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f3954900640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950153890 0x7f39501557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f3954900640 1 --2- >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3950155cf0 0x7f39501f9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f3954900640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f395006e5e0 con 0x7f393c005150 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f3954900640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 -- 0x7f3950164720 con 0x7f3950153890 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f3954900640 1 -- --> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] -- mon_getmap magic: 0 -- 0x7f395006e760 con 0x7f3950155cf0 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f3950151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394e59c640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f3950151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45648/0 (socket says 172.21.15.28:45648) 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394e59c640 1 -- 172.21.15.28:0/2492533019 learned_addr learned my addr 172.21.15.28:0/2492533019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394ed9d640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3950155cf0 0x7f39501f9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394dd9b640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950153890 0x7f39501557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394e59c640 1 -- 172.21.15.28:0/2492533019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3950155cf0 msgr2=0x7f39501f9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394e59c640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3950155cf0 0x7f39501f9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394e59c640 1 -- 172.21.15.28:0/2492533019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950153890 msgr2=0x7f39501557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394e59c640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950153890 0x7f39501557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.209+0000 7f394e59c640 1 -- 172.21.15.28:0/2492533019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3950156100 con 0x7f393c005150 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.210+0000 7f394e59c640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f3950151500 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3948007340 tx=0x7f3948007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.210+0000 7f3946ffd640 1 -- 172.21.15.28:0/2492533019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394800f940 con 0x7f393c005150 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.210+0000 7f3946ffd640 1 -- 172.21.15.28:0/2492533019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3948018070 con 0x7f393c005150 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.210+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3950157a50 con 0x7f393c005150 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.210+0000 7f3946ffd640 1 -- 172.21.15.28:0/2492533019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394800fd60 con 0x7f393c005150 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.210+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3950157d30 con 0x7f393c005150 2024-04-25T19:14:04.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.211+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3950050740 con 0x7f393c005150 2024-04-25T19:14:04.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.212+0000 7f3946ffd640 1 -- 172.21.15.28:0/2492533019 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f394801d460 con 0x7f393c005150 2024-04-25T19:14:04.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.212+0000 7f3946ffd640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f39300744e0 0x7f39300769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T19:14:04.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.212+0000 7f394dd9b640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f39300744e0 0x7f39300769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T19:14:04.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.212+0000 7f3946ffd640 1 -- 172.21.15.28:0/2492533019 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5421+0+0 (secure 0 0 0) 0x7f3948095180 con 0x7f393c005150 2024-04-25T19:14:04.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.213+0000 7f394dd9b640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f39300744e0 0x7f39300769f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f39500b4760 tx=0x7f3938008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T19:14:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.217+0000 7f3946ffd640 1 -- 172.21.15.28:0/2492533019 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3950156100 con 0x7f393c005150 2024-04-25T19:14:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.439+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39500d65b0 con 0x7f393c005150 2024-04-25T19:14:04.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.440+0000 7f3946ffd640 1 -- 172.21.15.28:0/2492533019 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) ==== 94+0+3866 (secure 0 0 0) 0x7f394805fa40 con 0x7f393c005150 2024-04-25T19:14:04.442 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-25T19:14:04.442 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5368,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1413023330","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1413023330},{"type":"v1","addr":"172.21.15.53:6834","nonce":1413023330}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":5568,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/568086068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":568086068},{"type":"v1","addr":"172.21.15.28:6835","nonce":568086068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5574,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2217485298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2217485298},{"type":"v1","addr":"172.21.15.53:6835","nonce":2217485298}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":21},{"gid":6123,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/4172444721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":4172444721},{"type":"v1","addr":"172.21.15.28:6837","nonce":4172444721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":34}],"filesystems":[]} 2024-04-25T19:14:04.442 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 34 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.441+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f39300744e0 msgr2=0x7f39300769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.441+0000 7f3954900640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f39300744e0 0x7f39300769f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f39500b4760 tx=0x7f3938008040 comp rx=0 tx=0).stop 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.441+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 msgr2=0x7f3950151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.441+0000 7f3954900640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f3950151500 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f3948007340 tx=0x7f3948007800 comp rx=0 tx=0).stop 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.442+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 shutdown_connections 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.442+0000 7f3954900640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:6832/2354405395,v1:172.21.15.28:6833/2354405395] conn(0x7f39300744e0 0x7f39300769f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.442+0000 7f3954900640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.53:3301/0,v1:172.21.15.53:6790/0] conn(0x7f3950155cf0 0x7f39501f9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.442+0000 7f3954900640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3950153890 0x7f39501557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.442+0000 7f3954900640 1 --2- 172.21.15.28:0/2492533019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f393c005150 0x7f3950151500 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.442+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 >> 172.21.15.28:0/2492533019 conn(0x7f39500b7ad0 msgr2=0x7f395006ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.442+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 shutdown_connections 2024-04-25T19:14:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-25T19:14:04.442+0000 7f3954900640 1 -- 172.21.15.28:0/2492533019 wait complete. 2024-04-25T19:14:04.455 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-25T19:14:04.455 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T19:14:04.455 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T19:14:04.455 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.465 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T19:14:04.465 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T19:14:04.465 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.517 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T19:14:04.517 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T19:14:04.517 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.529 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T19:14:04.529 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T19:14:04.530 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.582 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T19:14:04.582 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T19:14:04.583 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T19:14:04.583 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.615 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T19:14:04.615 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T19:14:04.615 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.682 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T19:14:04.682 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T19:14:04.682 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.749 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T19:14:04.750 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T19:14:04.750 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.814 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T19:14:04.814 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T19:14:04.814 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.851 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T19:14:04.851 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T19:14:04.852 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:04.933 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T19:14:04.933 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T19:14:04.933 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:05.005 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T19:14:05.005 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T19:14:05.005 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:05.075 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T19:14:05.075 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T19:14:05.075 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T19:14:05.075 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:05.102 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T19:14:05.102 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T19:14:05.102 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:05.144 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T19:14:05.144 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T19:14:05.145 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T19:14:05.145 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:05.156 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T19:14:05.156 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T19:14:05.156 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:05.198 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T19:14:05.198 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T19:14:05.198 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T19:14:05.250 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T19:14:05.251 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T19:14:05.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'unmatched fragstat' | egrep -v 'unmatched rstat' | egrep -v 'was unreadable, recreating it now' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v MDS_FAILED | egrep -v MDS_DAMAGE | head -n 1 2024-04-25T19:14:05.289 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:14:05.290 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T19:14:05.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:14:05.499 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T19:14:05.581 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:14:05.582 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T19:14:05.640 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:14:05.640 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T19:14:05.689 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:14:05.689 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T19:14:05.817 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:14:05.818 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T19:14:05.930 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:14:05.930 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T19:14:05.987 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:14:05.988 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T19:14:06.038 INFO:tasks.ceph:Archiving mon data... 2024-04-25T19:14:06.039 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/data/mon.a.tgz 2024-04-25T19:14:06.039 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-04-25T19:14:06.055 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.W60LbB1pdB 2024-04-25T19:14:06.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.W60LbB1pdB 2024-04-25T19:14:06.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.W60LbB1pdB 2024-04-25T19:14:06.281 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.W60LbB1pdB is 347KB 2024-04-25T19:14:06.380 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.W60LbB1pdB 2024-04-25T19:14:06.396 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/data/mon.b.tgz 2024-04-25T19:14:06.396 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-04-25T19:14:06.415 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.icX6KQ3NpL 2024-04-25T19:14:06.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.icX6KQ3NpL 2024-04-25T19:14:06.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.icX6KQ3NpL 2024-04-25T19:14:06.666 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.icX6KQ3NpL is 460KB 2024-04-25T19:14:06.773 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.icX6KQ3NpL 2024-04-25T19:14:06.789 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/data/mon.c.tgz 2024-04-25T19:14:06.789 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-04-25T19:14:06.844 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.fDvvz5Ct50 2024-04-25T19:14:06.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.fDvvz5Ct50 2024-04-25T19:14:06.999 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.fDvvz5Ct50 2024-04-25T19:14:07.082 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.fDvvz5Ct50 is 460KB 2024-04-25T19:14:07.151 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.fDvvz5Ct50 2024-04-25T19:14:07.166 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T19:14:07.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T19:14:07.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T19:14:07.254 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T19:14:07.286 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T19:14:07.286 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T19:14:07.287 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/remote/smithi028/crash 2024-04-25T19:14:07.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T19:14:07.323 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/remote/smithi053/crash 2024-04-25T19:14:07.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T19:14:07.350 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T19:14:07.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:14:07.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T19:14:07.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T19:14:07.470 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T19:14:07.495 INFO:tasks.ceph:Compressing logs... 2024-04-25T19:14:07.495 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:14:07.513 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T19:14:07.536 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42593.log 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34497.log 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34748.log 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34497.log.gz 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34748.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34748.log.gz 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T19:14:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mds.b.log: 91.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34708.log 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34708.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.40314.log 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34708.log.gz 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.1.40314.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.3.40712.log 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.1.40314.log.gz 2024-04-25T19:14:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.6.log: 92.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T19:14:07.582 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34628.log 2024-04-25T19:14:07.582 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.3.40712.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34581.log 2024-04-25T19:14:07.582 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.3.40712.log.gz 2024-04-25T19:14:07.582 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34537.log 2024-04-25T19:14:07.582 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34581.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.34628.log.gz 2024-04-25T19:14:07.582 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34581.log.gz 2024-04-25T19:14:07.582 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34457.log 2024-04-25T19:14:07.582 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34537.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34537.log.gz 2024-04-25T19:14:07.595 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T19:14:07.595 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34457.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T19:14:07.595 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34457.log.gz 2024-04-25T19:14:07.595 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5/var/log/ceph/ceph-mds.d.log: --verbose -- /var/log/ceph/ceph-client.admin.34668.log 2024-04-25T19:14:07.604 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.b.log: 92.8% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T19:14:07.604 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.34668.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34668.log.gz 2024-04-25T19:14:07.635 INFO:teuthology.orchestra.run.smithi053.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T19:14:07.669 INFO:teuthology.orchestra.run.smithi053.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53151.log 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52177.log 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43727.log 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42593.log.gz 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53151.log.gz 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47342.log 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47180.log 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52177.log.gz 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47342.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43727.log.gz 2024-04-25T19:14:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54230.log 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47342.log.gz 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50172.log 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54230.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47180.log.gz 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50263.log 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50172.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54230.log.gz 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45473.log 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46466.log 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7%/var/log/ceph/ceph-client.admin.45473.log: -- replaced with /var/log/ceph/ceph-client.admin.50263.log.gz 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47552.log 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.50172.log.gz 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45473.log.gz 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43148.log 2024-04-25T19:14:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46466.log: /var/log/ceph/ceph-client.admin.47552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45851.log 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46466.log.gz 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47552.log.gz 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43601.log 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43148.log.gz 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54398.log 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43601.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45851.log.gz 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48280.log 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54398.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43601.log.gz 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54692.log 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48280.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54398.log.gz 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43147.log 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48280.log.gz 2024-04-25T19:14:07.687 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42811.log 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43147.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54692.log.gz 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54440.log 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43147.log.gz 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47138.log 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42811.log.gz 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52732.log 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47138.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.54440.log.gz 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46508.log 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47138.log.gz 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51273.log 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52732.log.gz 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53319.log 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51273.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46508.log.gz 2024-04-25T19:14:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52219.log 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48403.log 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.51273.log.gz 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53319.log.gz 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46061.log 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52219.log.gz 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50914.log 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46061.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.48403.log.gz --verbose 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr: -- /var/log/ceph/ceph-client.admin.42257.log 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46061.log.gz 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40022.log 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50914.log.gz 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40368.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.42816.log -- replaced with /var/log/ceph/ceph-client.admin.42257.log.gz 2024-04-25T19:14:07.689 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53109.log 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42816.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40022.log.gz 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53445.log 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53109.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42816.log.gz 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54734.log 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53109.log.gz 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54482.log 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54734.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53445.log.gz 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40452.log 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54482.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54734.log.gzgzip 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52942.log 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40452.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54482.log.gz 2024-04-25T19:14:07.690 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48996.log 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40452.log.gz 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54566.log 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52942.log.gz 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47888.log 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54566.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48996.log.gz 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49290.log 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54566.log.gz 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48782.log 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47888.log: /var/log/ceph/ceph-client.admin.49290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41467.log 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47888.log.gz 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48782.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49290.log.gz 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46550.log 2024-04-25T19:14:07.691 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48782.log.gz 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40199.log 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48740.log 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46550.log.gz/var/log/ceph/ceph-client.admin.40199.log: 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52093.log 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40199.log.gz 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34384.log 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42817.log 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48740.log.gz 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52093.log.gz 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34384.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.41467.log.gz 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45725.log 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.34384.log.gz 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46145.log 2024-04-25T19:14:07.692 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48826.log 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42817.log.gz 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46145.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45725.log.gz -5 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45977.log 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48826.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46145.log.gz 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43153.log 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48826.log.gz 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.log: gzip -5 89.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.45977.log.gz -- 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.45221.log 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54314.log 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45221.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.47972.log -- replaced with /var/log/ceph/ceph-client.admin.43153.log.gz 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr: 85.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T19:14:07.693 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45221.log.gz 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53361.log 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54314.log: /var/log/ceph/ceph-client.admin.47972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45641.log 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54314.log.gz 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47972.log.gz 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49332.log 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45641.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53361.log.gz 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44245.log 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49332.log: gzip -5 88.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45641.log.gz 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.42812.log 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41743.log 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.49332.log.gz 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52480.log 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44245.log.gz 2024-04-25T19:14:07.694 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42812.log.gz 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46271.log 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52480.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41743.log.gz 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49806.log 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52480.log.gz 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46271.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45515.log 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46271.log.gz 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.2.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49806.log.gz 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41701.log 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52858.log 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41701.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45515.log.gz 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45347.log 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41701.log.gz 2024-04-25T19:14:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42509.log 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45347.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52858.log.gz 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43685.log 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42509.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45347.log.gz 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40326.log 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42509.log.gz 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46592.log 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40326.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43685.log.gz 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47762.log 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46592.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40326.log.gz 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50872.log 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46592.log.gz 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42635.log 2024-04-25T19:14:07.696 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42551.log 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47762.log.gz 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50872.log.gz 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49848.log 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42635.log.gz 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42551.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42551.log.gz 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45003.log 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50536.log 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49848.log.gz 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.2.45003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.45003.log.gz 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51147.log 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53692.log 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51147.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50536.log.gz 2024-04-25T19:14:07.697 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54146.log 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47804.log 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49890.log 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54146.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53692.log.gz 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43895.log 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54146.log.gz 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47804.log.gz 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54356.log 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.51147.log.gz 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49890.log.gz 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42814.log 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43895.log: /var/log/ceph/ceph-client.admin.54356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34751.log 2024-04-25T19:14:07.698 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43895.log.gz 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54356.log.gz 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54650.log 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42814.log.gz 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54650.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34751.log.gz 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47300.log 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54650.log.gz 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41969.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47300.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44021.log 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47300.log.gz 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50937.log 2024-04-25T19:14:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43152.log 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50937.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44021.log.gz 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50746.log 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52303.log 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43152.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.50937.log.gz 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42215.log 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43152.log.gz 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50746.log.gz 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40241.log 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52303.log.gz 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45935.log 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40241.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42215.log.gz 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45935.log: gzip -5 --verbose 89.3% -- -- replaced with /var/log/ceph/ceph-client.admin.40241.log.gz /var/log/ceph/ceph-client.admin.53506.log 2024-04-25T19:14:07.700 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45935.log.gz 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46838.log 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41046.log: /var/log/ceph/ceph-client.admin.53506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42383.log 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46838.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53506.log.gz 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41927.log 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42383.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46838.log.gz 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47930.log 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42383.log.gz 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43643.log 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41927.log.gz 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41827.log 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47930.log.gz 2024-04-25T19:14:07.701 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54103.log 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43643.log.gz 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.41827.log: -5 --verbose -- /var/log/ceph/ceph-client.0.51927.log 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51393.log 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41827.log.gz 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.51927.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54103.log.gz 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.51927.log.gz 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47720.log 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51393.log: /var/log/ceph/ceph-mds.c.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51393.log.gz 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51189.log 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40686.log 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47720.log: /var/log/ceph/ceph-client.admin.51189.log: gzip -5 --verbose -- 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47720.log.gz 2024-04-25T19:14:07.702 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.50662.log 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52984.log 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40686.log.gz 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45557.log 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.51189.log.gz 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53277.log 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50662.log.gz 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.45557.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45305.log 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52984.log.gz 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53277.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45557.log.gz 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53277.log.gz 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48200.log 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53882.log 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45305.log.gz 2024-04-25T19:14:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44564.log 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53882.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48200.log.gz 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54608.log 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53882.log.gz 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.44564.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.44564.log.gz 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54608.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52648.log 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54608.log.gz 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50450.log 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46634.log 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50450.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52648.log.gz 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43391.log 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50450.log.gz 2024-04-25T19:14:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47510.log 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46634.log: /var/log/ceph/ceph-client.admin.43391.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46634.log.gz 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43391.log.gz 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52774.log 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47510.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50979.log 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52774.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47510.log.gz 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42299.log 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52774.log.gz 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43937.log 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42299.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42299.log.gz 90.8% -- replaced with /var/log/ceph/ceph-client.admin.50979.log.gz 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46229.log 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48912.log 2024-04-25T19:14:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43937.log: /var/log/ceph/ceph-client.admin.46229.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43937.log.gz 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46229.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46424.log 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51631.log 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48912.log: /var/log/ceph/ceph-client.admin.46424.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48912.log.gz 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43517.log 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46424.log.gz 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45809.log 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43517.log: /var/log/ceph/ceph-client.admin.51631.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43517.log.gz 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51631.log.gz 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54061.log 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45893.log 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45809.log.gz 2024-04-25T19:14:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52396.log 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45893.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54061.log.gz 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45893.log.gz 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45137.log 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50788.log 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45137.log: /var/log/ceph/ceph-client.admin.52396.log: 88.3% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52396.log.gz 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45137.log.gz 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40540.log 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46880.log 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50788.log: /var/log/ceph/ceph-client.admin.40540.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50788.log.gz 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.40540.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.45263.log 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40854.log 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46880.log: /var/log/ceph/ceph-client.admin.45263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52900.log 2024-04-25T19:14:07.707 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46880.log.gz 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40854.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45263.log.gz 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40854.log.gz 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49206.log 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49932.log 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52900.log.gz 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49206.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49206.log.gz 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45431.log 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48954.log 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49932.log.gz 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52690.log 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48954.log: /var/log/ceph/ceph-client.admin.45431.log: 88.4% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45431.log.gz 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48954.log.gz 2024-04-25T19:14:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51021.log 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41785.log 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52690.log: /var/log/ceph/ceph-client.admin.51021.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52690.log.gz 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46712.log 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.51021.log.gz 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51481.log 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46712.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41785.log.gz 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46712.log.gz 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42467.log 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50305.log 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51481.log: /var/log/ceph/ceph-client.admin.42467.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51481.log.gz 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49398.log 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42467.log.gz 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51105.log 2024-04-25T19:14:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49398.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50305.log.gz 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49398.log.gz 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40728.log 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51105.log: /var/log/ceph/ceph-client.admin.40728.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40728.log.gz 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42804.log 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.51105.log.gz 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51340.log 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42804.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42804.log.gz 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43811.log 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47846.log 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51340.log.gz 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48032.log 2024-04-25T19:14:07.710 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4%/var/log/ceph/ceph-client.admin.47846.log: -- replaced with /var/log/ceph/ceph-client.admin.43811.log.gz 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34648.log 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47846.log.gz 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50051.log 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.34648.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48032.log.gz 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34648.log.gz 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53966.log 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42053.log 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50051.log: /var/log/ceph/ceph-client.admin.53966.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50051.log.gz 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52606.log 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53966.log.gz 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51435.log 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52606.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42053.log.gz 2024-04-25T19:14:07.711 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52606.log.gz 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42425.log 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50347.log 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51435.log: /var/log/ceph/ceph-client.admin.42425.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51435.log.gz 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53924.log 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42425.log.gz 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46754.log 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53924.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50347.log.gz 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53924.log.gz 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47594.log 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51697.log 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46754.log: /var/log/ceph/ceph-client.admin.47594.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46754.log.gz 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48870.log 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47594.log.gz 2024-04-25T19:14:07.712 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40812.log 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48870.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51697.log.gz 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48870.log.gz 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53403.log 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50830.log 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40812.log: /var/log/ceph/ceph-client.admin.53403.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40812.log.gz 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46922.log 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53403.log.gz 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49080.log 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46922.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50830.log.gz 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46922.log.gz 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43146.log 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49080.log: /var/log/ceph/ceph-client.admin.43146.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49080.log.gz 2024-04-25T19:14:07.713 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41659.log 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43146.log.gz 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43150.log 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.41659.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41659.log.gz 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45683.log 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40624.log 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43150.log: /var/log/ceph/ceph-client.admin.45683.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43150.log.gz 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54008.log 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45683.log.gz 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42815.log 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54008.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40624.log.gz 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54008.log.gz 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50009.log 2024-04-25T19:14:07.714 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47096.log 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42815.log: /var/log/ceph/ceph-client.admin.50009.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42815.log.gz 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50009.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46313.log 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43853.log 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8%/var/log/ceph/ceph-client.admin.46313.log: -- replaced with /var/log/ceph/ceph-client.admin.47096.log.gz 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48698.log 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46313.log.gz 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43853.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43853.log.gz 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50620.log 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48614.log 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48698.log: /var/log/ceph/ceph-client.admin.50620.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48698.log.gz 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50620.log.gz 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47678.log 2024-04-25T19:14:07.715 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5/var/log/ceph/ceph-client.admin.48614.log: --verbose -- /var/log/ceph/ceph-client.admin.43349.log 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48614.log.gz 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51523.log 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47678.log.gz 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47636.log 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51523.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43349.log.gz 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53235.log 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47636.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.51523.log.gz 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47636.log.gz 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50494.log 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47426.log 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50494.log: /var/log/ceph/ceph-client.admin.53235.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53235.log.gz 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50494.log.gz 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52135.log 2024-04-25T19:14:07.716 INFO:teuthology.orchestra.run.smithi028.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43979.log 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47426.log: /var/log/ceph/ceph-client.admin.52135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48530.log 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43979.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47426.log.gz 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52135.log.gz 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28326.log 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43979.log.gz 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46019.log 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46796.log 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48530.log.gz 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph.tmp-client.admin.28326.log: 0.0%/var/log/ceph/ceph-client.admin.46019.log: -- replaced with /var/log/ceph/ceph.tmp-client.admin.28326.log.gz 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42095.log 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0%/var/log/ceph/ceph-client.admin.46796.log: -- replaced with /var/log/ceph/ceph-client.admin.46019.log.gz 2024-04-25T19:14:07.717 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40582.log 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46796.log.gz 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52522.log 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53193.log 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42095.log.gz 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40582.log: /var/log/ceph/ceph-client.admin.52522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48656.log 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40582.log.gz 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53193.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52522.log.gz 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49248.log 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53193.log.gz 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54794.log 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48656.log.gz 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39929.log 2024-04-25T19:14:07.718 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49248.log.gz 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51589.log 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54794.log.gz 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.39929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54876.log 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2%/var/log/ceph/ceph-client.admin.51589.log: -- replaced with /var/log/ceph/ceph-client.admin.39929.log.gz 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52438.log 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51589.log.gz 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54272.log 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54876.log: /var/log/ceph/ceph-client.admin.52438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48572.log 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54272.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52438.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54876.log.gz 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54272.log.gz 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45389.log 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47012.log 2024-04-25T19:14:07.719 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45389.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48572.log.gz 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53566.log 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1%/var/log/ceph/ceph-client.admin.47012.log: -- replaced with /var/log/ceph/ceph-client.admin.45389.log.gz 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43143.log 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47012.log.gz 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50578.log 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53566.log: /var/log/ceph/ceph-client.admin.43143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48361.log 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.50578.log: 88.6% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43143.log.gz 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53566.log.gz 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50578.log.gz 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44063.log 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2%/var/log/ceph/ceph-client.admin.40410.log: -- replaced with /var/log/ceph/ceph-client.admin.48361.log.gz 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47468.log 2024-04-25T19:14:07.720 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48445.log 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46187.log 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47468.log: /var/log/ceph/ceph-client.admin.48445.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47468.log.gz 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44063.log.gz 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40980.log 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48445.log.gz 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48488.log 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46187.log: /var/log/ceph/ceph-client.admin.40980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52816.log 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr: 87.5%/var/log/ceph/ceph-client.admin.48488.log: -- replaced with /var/log/ceph/ceph-client.admin.46187.log.gz 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48158.log 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48488.log.gz 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40980.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42011.log 2024-04-25T19:14:07.721 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52816.log: /var/log/ceph/ceph-client.admin.48158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44699.log 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52816.log.gz 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48158.log.gz 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42011.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.44699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54524.log 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42011.log.gz 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-mds.a.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44699.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48074.log 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40896.log 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48074.log: /var/log/ceph/ceph-client.admin.54524.log: 88.0% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54524.log.gz 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48074.log.gz 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53025.log 2024-04-25T19:14:07.722 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49038.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40896.log.gz 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42341.log 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53025.log.gz/var/log/ceph/ceph-client.admin.49038.log: 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43769.log 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50390.log 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49038.log.gz 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42341.log: gzip 88.4% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.42341.log.gz -- 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr: /var/log/ceph/ceph-client.admin.43433.log 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43769.log: gzip/var/log/ceph/ceph-client.admin.50390.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48116.log 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43433.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50390.log.gz 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43769.log.gz 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40938.log 2024-04-25T19:14:07.723 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43433.log.gz 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.48116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45599.log 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49164.log 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48116.log.gz 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40938.log: /var/log/ceph/ceph-client.admin.45599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53067.log 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49164.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40938.log.gz 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45599.log.gz 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49164.log.gz 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41869.log 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.53067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47222.log 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53067.log.gz/var/log/ceph/ceph-client.admin.41869.log: 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47384.log 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41869.log.gz/var/log/ceph/ceph-client.admin.47222.log: 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45179.log 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4%/var/log/ceph/ceph-client.admin.47384.log: -- replaced with /var/log/ceph/ceph-client.admin.47222.log.gz 2024-04-25T19:14:07.724 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43475.log 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47384.log.gz 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.45179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53650.log 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45179.log.gz 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40118.log 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43475.log: /var/log/ceph/ceph-client.admin.53650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42137.log 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40118.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43475.log.gz 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53650.log.gz 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40118.log.gz 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52261.log 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54834.log 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42137.log.gz 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49764.log 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.52261.log.gz 2024-04-25T19:14:07.725 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49586.log 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54834.log: /var/log/ceph/ceph-client.admin.49764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52564.log 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.0.49586.log: 88.2% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49764.log.gz 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54834.log.gz 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.0.49586.log.gz 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51231.log 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.52564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40284.log 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:gzip/var/log/ceph/ceph-client.admin.51231.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43559.log 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52564.log.gz 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40284.log: 91.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40496.log 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51231.log.gz 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43559.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40284.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53608.log 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.726 INFO:teuthology.orchestra.run.smithi028.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43559.log.gz 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.40496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43151.log 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7%/var/log/ceph/ceph-client.admin.53608.log: -- replaced with /var/log/ceph/ceph-client.admin.40496.log.gz 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49122.log 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53608.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54188.log 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.43151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46103.log 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.49122.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43151.log.gz 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.54188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42805.log 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49122.log.gz 87.5% -- replaced with /var/log/ceph/ceph-client.admin.54188.log.gz 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.46103.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46103.log.gz 2024-04-25T19:14:07.727 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47054.log 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.42805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50704.log 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.47054.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42805.log.gz 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51063.log 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47054.log.gz 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5/var/log/ceph/ceph-client.admin.50704.log: --verbose -- /var/log/ceph/ceph-client.admin.45767.log 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51063.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51739.log 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50704.log.gz 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr: 90.6%/var/log/ceph/ceph-client.admin.45767.log: -- replaced with /var/log/ceph/ceph-client.admin.51063.log.gz 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/ceph-client.admin.51739.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45767.log.gz 2024-04-25T19:14:07.728 INFO:teuthology.orchestra.run.smithi028.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51739.log.gz 2024-04-25T19:14:07.729 INFO:teuthology.orchestra.run.smithi053.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T19:14:07.934 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T19:14:08.264 INFO:teuthology.orchestra.run.smithi028.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T19:14:08.280 INFO:teuthology.orchestra.run.smithi053.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T19:14:08.309 INFO:teuthology.orchestra.run.smithi053.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T19:14:08.324 INFO:teuthology.orchestra.run.smithi028.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T19:14:08.347 INFO:teuthology.orchestra.run.smithi053.stderr: 93.8% 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T19:14:08.347 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-25T19:14:08.349 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-04-25T19:14:08.349 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m0.800s 2024-04-25T19:14:08.349 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m3.231s 2024-04-25T19:14:08.349 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m0.126s 2024-04-25T19:14:08.352 INFO:teuthology.orchestra.run.smithi028.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T19:14:08.393 INFO:teuthology.orchestra.run.smithi028.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T19:14:08.395 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-25T19:14:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:real 0m0.869s 2024-04-25T19:14:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:user 0m3.517s 2024-04-25T19:14:08.395 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m0.289s 2024-04-25T19:14:08.395 INFO:tasks.ceph:Archiving logs... 2024-04-25T19:14:08.395 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/remote/smithi028/log 2024-04-25T19:14:08.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:14:09.183 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/remote/smithi053/log 2024-04-25T19:14:09.184 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T19:14:09.677 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T19:14:09.690 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T19:14:09.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:14:09.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T19:14:09.771 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T19:14:09.783 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T19:14:09.783 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:14:09.786 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T19:14:09.800 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:14:09.803 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-04-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:14:09.804 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 6 377 55 +897us[ +894us] +/- 52ms 2024-04-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 -1503us[-1506us] +/- 75ms 2024-04-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 54 +188us[ +188us] +/- 85ms 2024-04-25T19:14:09.805 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-04-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv01.front.sepia.ceph.com 3 6 377 53 +915us[ +927us] +/- 52ms 2024-04-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 55 -1499us[-1488us] +/- 75ms 2024-04-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 +195us[ +206us] +/- 85ms 2024-04-25T19:14:09.808 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T19:14:09.809 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T19:14:09.821 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T19:14:09.822 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T19:14:09.837 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:14:09.923 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T19:14:09.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:14:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:10.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T19:14:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:10.051 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T19:14:10.064 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T19:14:10.079 INFO:teuthology.task.internal:Duration was 533.360387 seconds 2024-04-25T19:14:10.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T19:14:10.095 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T19:14:10.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:14:10.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T19:14:10.136 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:14:10.138 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T19:14:10.465 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T19:14:10.465 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-04-25T19:14:10.466 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:14:10.529 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-04-25T19:14:10.530 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T19:14:10.557 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T19:14:10.558 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:14:10.572 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T19:14:10.626 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T19:14:10.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:14:10.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T19:14:10.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T19:14:10.782 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T19:14:10.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:14:10.807 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T19:14:10.831 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T19:14:10.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:10.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:10.869 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-04-25T19:14:10.894 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-04-25T19:14:10.908 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:10.937 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T19:14:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T19:14:10.963 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T19:14:10.976 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T19:14:10.977 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/remote/smithi028 2024-04-25T19:14:10.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:14:11.010 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673302/remote/smithi053 2024-04-25T19:14:11.011 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T19:14:11.044 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T19:14:11.044 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:14:11.048 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T19:14:11.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T19:14:11.106 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T19:14:11.107 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T19:14:11.122 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T19:14:11.122 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:14:11.124 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T19:14:11.136 INFO:teuthology.orchestra.run.smithi028.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:14 /home/ubuntu/cephtest 2024-04-25T19:14:11.149 INFO:teuthology.orchestra.run.smithi053.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 19:14 /home/ubuntu/cephtest 2024-04-25T19:14:11.149 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T19:14:11.162 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T19:14:11.196 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} duration: 533.3603870868683 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T19:14:11.196 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T19:14:11.276 INFO:teuthology.run:pass